From 42e4e50f5be654b812939732114f0724865cdc78 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Wed, 4 Jan 2023 13:45:37 +0100 Subject: [PATCH 001/382] Matrix Authentication Support for Jitsi This extends the collection with support for seamless authentication at the Jitsi server using Matrix OpenID. 1. New role for installing the [Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service) 2. Changes to Jitsi role: Installing Jitsi Prosody Mods and configuring Jitsi Auth 3. Changes to Jitsi and nginx-proxy roles: Serving .well-known/element/jitsi from jitsi.DOMAIN 4. We updated the Jitsi documentation on authentication and added documentation for the user verification service. --- docs/configuring-playbook-jitsi.md | 47 ++++++- ...ring-playbook-user-verification-service.md | 116 ++++++++++++++++++ group_vars/matrix_servers | 61 ++++++++- playbooks/matrix.yml | 1 + roles/custom/matrix-jitsi/defaults/main.yml | 36 +++++- roles/custom/matrix-jitsi/tasks/main.yml | 6 + .../tasks/self_check_matrix_auth.yml | 62 ++++++++++ .../tasks/setup_jitsi_prosody_install.yml | 46 ++++++- .../setup_jitsi_auth_internal.yml | 17 +++ .../tasks/util/setup_jitsi_auth.yml | 42 ------- .../util/setup_jitsi_auth_uvs_install.yml | 13 ++ .../util/setup_jitsi_auth_uvs_uninstall.yml | 26 ++++ .../setup_jitsi_prosody_post_setup_hooks.yml | 49 ++++++++ .../matrix-jitsi/tasks/validate_config.yml | 15 ++- .../matrix-jitsi/templates/prosody/env.j2 | 8 +- .../templates/web/custom-config.js.j2 | 4 + .../matrix-nginx-proxy/defaults/main.yml | 2 + .../nginx/conf.d/matrix-jitsi.conf.j2 | 11 +- .../defaults/main.yml | 81 ++++++++++++ .../handlers/main.yml | 6 + .../tasks/main.yml | 24 ++++ .../tasks/setup_install.yml | 42 +++++++ .../tasks/setup_uninstall.yml | 35 ++++++ .../templates/.env.j2 | 14 +++ ...atrix-user-verification-service.service.j2 | 42 +++++++ 25 files changed, 747 insertions(+), 59 deletions(-) create mode 100644 docs/configuring-playbook-user-verification-service.md create mode 100644 roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml create mode 100644 roles/custom/matrix-jitsi/tasks/util/prosody_post_setup_hooks/setup_jitsi_auth_internal.yml delete mode 100644 roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth.yml create mode 100644 roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml create mode 100644 roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml create mode 100644 roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml create mode 100644 roles/custom/matrix-user-verification-service/defaults/main.yml create mode 100644 roles/custom/matrix-user-verification-service/handlers/main.yml create mode 100644 roles/custom/matrix-user-verification-service/tasks/main.yml create mode 100644 roles/custom/matrix-user-verification-service/tasks/setup_install.yml create mode 100644 roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml create mode 100644 roles/custom/matrix-user-verification-service/templates/.env.j2 create mode 100644 roles/custom/matrix-user-verification-service/templates/systemd/matrix-user-verification-service.service.j2 diff --git a/docs/configuring-playbook-jitsi.md b/docs/configuring-playbook-jitsi.md index a9d3c2de..f51f8fc1 100644 --- a/docs/configuring-playbook-jitsi.md +++ b/docs/configuring-playbook-jitsi.md @@ -39,7 +39,17 @@ By default the Jitsi Meet instance does not require any kind of login and is ope If you're fine with such an open Jitsi instance, please skip to [Apply changes](#apply-changes). -If you would like to control who is allowed to open meetings on your new Jitsi instance, then please follow this step to enable Jitsi's authentication and guests mode. With authentication enabled, all meeting rooms have to be opened by a registered user, after which guests are free to join. If a registered host is not yet present, guests are put on hold in individual waiting rooms. +If you would like to control who is allowed to open meetings on your new Jitsi instance, then please follow the following steps to enable Jitsi's authentication and optionally guests mode. +Currently, there are three supported authentication modes: 'internal' (default), 'matrix' and 'ldap'. + +**Note:** Authentication is not tested via the playbook's self-checks. +We therefore recommend that you manually verify if authentication is required by jitsi. +For this, try to manually create a conference on jitsi.DOMAIN in your browser. + +### Authenticate using Jitsi accounts (Auth-Type 'internal') +The default authentication mechanism is 'internal' auth, which requires jitsi-accounts to be setup and is the recommended setup, as it also works in federated rooms. +With authentication enabled, all meeting rooms have to be opened by a registered user, after which guests are free to join. +If a registered host is not yet present, guests are put on hold in individual waiting rooms. Add these lines to your `inventory/host_vars/matrix.DOMAIN/vars.yml` configuration: @@ -53,20 +63,35 @@ matrix_jitsi_prosody_auth_internal_accounts: password: "another-password" ``` -**Caution:** Accounts added here and subsquently removed will not be automatically removed from the Prosody server until user account cleaning is integrated into the playbook. +**Caution:** Accounts added here and subsequently removed will not be automatically removed from the Prosody server until user account cleaning is integrated into the playbook. **If you get an error** like this: "Error: Account creation/modification not supported.", it's likely that you had previously installed Jitsi without auth/guest support. In such a case, you should look into [Rebuilding your Jitsi installation](#rebuilding-your-jitsi-installation). +### Authenticate using Matrix OpenID (Auth-Type 'matrix') -### (Optional) LDAP authentication +**Attention: Probably breaks jitsi in federated rooms and does not allow sharing conference links with guests.** -The default authentication mode of Jitsi is `internal`, however LDAP is also supported. An example LDAP configuration could be: +Using this authentication type require a [Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service). +By default, this playbook creates and configures a user-verification-service to run locally, see [configuring-user-verification-service](configuring-playbook-user-verification-service.md). + +To enable set this configuration at host level: + +```yaml +matrix_jitsi_enable_auth: true +matrix_jitsi_auth_type: "matrix" +``` + +For more information see also [https://github.com/matrix-org/prosody-mod-auth-matrix-user-verification](https://github.com/matrix-org/prosody-mod-auth-matrix-user-verification). + +### Authenticate using LDAP (Auth-Type 'ldap') + +An example LDAP configuration could be: ```yaml matrix_jitsi_enable_auth: true matrix_jitsi_auth_type: ldap matrix_jitsi_ldap_url: "ldap://ldap.DOMAIN" -matrix_jitsi_ldap_base: "OU=People,DC=DOMAIN +matrix_jitsi_ldap_base: "OU=People,DC=DOMAIN" #matrix_jitsi_ldap_binddn: "" #matrix_jitsi_ldap_bindpw: "" matrix_jitsi_ldap_filter: "uid=%u" @@ -200,7 +225,19 @@ matrix_nginx_proxy_proxy_jitsi_additional_jvbs: Applied together this will allow you to provision extra JVB instances which will register themselves with the prosody service and be available for jicofo to route conferences too. +## (Optional) Enable Gravatar + +In the default Jisti Meet configuration, gravatar.com is enabled as an avatar service. This results in third party request leaking data to gravatar. +Since element already sends the url of configured Matrix avatars to Jitsi, we disabled gravatar. + +To enable Gravatar set: + +```yaml +matrix_jitsi_disable_gravatar: false +``` +**Beware:** This leaks information to a third party, namely the Gravatar-Service (unless configured otherwise: gravatar.com). +Besides metadata, this includes the matrix user_id and possibly the room identifier (via `referrer` header). ## Apply changes diff --git a/docs/configuring-playbook-user-verification-service.md b/docs/configuring-playbook-user-verification-service.md new file mode 100644 index 00000000..f3d3aa6e --- /dev/null +++ b/docs/configuring-playbook-user-verification-service.md @@ -0,0 +1,116 @@ +# Setting up Matrix User Verification Service (optional) + +**[Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service) (hereafter: UVS) can only be installed after Matrix services are installed and running.** +If you're just installing Matrix services for the first time, please continue with the [Configuration](configuring-playbook.md) / [Installation](installing.md) flow and come back here later. + +Currently, the main purpose of this role is to allow Jitsi to authenticate matrix users and check if they are authorized to join a conference. Please refer to the documentation of the [Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service) to understand how it works. + +**Note**: enabling Matrix User Verification Service, means that the `openid` API endpoints will be exposed on the Matrix Federation port (usually `8448`), even if [federation](configuring-playbook-federation.md) is disabled. + +If the Jitsi server is also configured by this collection, all plugging of variables and secrets is handled in `group_vars/matrix_servers`. + +__Some general concepts of UVS may be helpful to understand the rest, so here they are:__ + +UVS can be used to verify two claims: + +* (A) Whether a given OpenID token is valid for a given server and +* (B) whether a user is member of a given room and the corresponding PowerLevel + +Verifying an OpenID token id done by finding the corresponding Homeserver via '.well-known/matrix/server' for the given domain. +The configured `matrix_user_verification_service_uvs_homeserver_url` does **not** factor into this. +By default, this collection only checks against `matrix_server_fqn_matrix`. +Therefore, the request will be made against the public openid API for `matrix_server_fqn_matrix`. + +Verifying RoomMembership and PowerLevel is done against `matrix_user_verification_service_uvs_homeserver_url` which is by default done via the docker network. +UVS will verify the validity of the token beforehand though. + +## Prerequisites + +In order to use UVS, an admin token for the configured homeserver must be supplied. For now this means configuring Synapse and creating the token before installing UVS. + +## Enable + +[Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service) installation is disabled by default unless required by Jitsi (see group_vars/matrix_servers). +You can enable it in your configuration file (`inventory/host_vars/matrix./vars.yml`): + +```yaml +matrix_user_verification_service_enabled: true +``` + +## Configuration + +The only required configuration variable is `matrix_user_verification_service_uvs_access_token` (see below). + +For a list of all configuration options see the role defaults [`roles/matrix-user-verification-service/defaults/main.yml`](../roles/custom/matrix-user-verification-service/defaults/main.yml). +But be aware of all the plugging happening in `group_vars/matrix_servers`. + +In the default configuration, the UVS Server is only reachable via the docker network, which is fine if e.g. Jitsi is also running in a container on the host. +However, it is possible to expose UVS via setting `matrix_user_verification_service_container_http_host_bind_port`. Be aware that the normally used port (3000) may collide with Grafana. + +### Access token + +The Synapse Access Token is used to verify RoomMembership and PowerLevel against the configured homeserver_url (which is plugged in group_vars). + +We recommend that you create a dedicated Matrix user for uvs (`uvs` is a good username). +Follow our [Registering users](registering-users.md) guide to register a user with administration privileges. + +You are required to specify an access token (belonging to this new user) for UVS to work. +To get an access token for the UVS user, you can follow the documentation on [how to do obtain an access token](obtaining-access-tokens.md). + +**Access tokens are sensitive information. Do not include them in any bug reports, messages, or logs. Do not share the access token with anyone.** + +```yaml +matrix_user_verification_service_uvs_access_token: "YOUR ACCESS TOKEN HERE" +``` + +### (Optional) Auth Token + +It is possible to set an API Auth Token to restrict access to the UVS. If this is set, anyone making a request to UVS must provide it via the header "Authorization: Bearer TOKEN" + +By default, the token will be derived from `matrix_homeserver_generic_secret_key` in `group_vars/matrix_servers`. +To set your own Token, simply put the following in your host_vars. + +```yaml +matrix_user_verification_service_uvs_auth_token: "TOKEN" +``` + +In case Jitsi is also managed by this collection and 'matrix' authentication in Jitsi is enabled, this collection will automatically configure Jitsi to use the configured auth token. + +### (Optional) Federation + +In theory (however currently untested), UVS can handle federation. Simply set: + +```yaml +matrix_user_verification_service_uvs_openid_verify_server_name: ~ +``` + +using host_vars to override the group_vars. + +This will instruct UVS to verify the OpenID token against any domain given in a request. +Homeserver discovery is done via '.well-known/matrix/server' of the given domain. + +## Installation + +After these variables have been set, please run the following command to re-run setup and to restart UVS: + +``` +ansible-playbook -i inventory/hosts setup.yml --tags=setup-matrix-user-verification-service,start +``` + +## Logging + +The configuration variable `UVS_LOG_LEVEL` can be set to: +- warning +- info +- debug + +## TLS Certificate Checking +If the matrix Homeserver does not provide a valid TLS certificate, UVS will fail with the following error message: + +> message: 'No response received: [object Object]', + +This also applies to self-signed and let's encrypt staging certificates. + +To disable certificate validation altogether (INSECURE! Not suitable for production use!) set: `NODE_TLS_REJECT_UNAUTHORIZED=0` + +Alternatively, it is possible to inject your own CA certificates into the container by mounting a PEM file with additional trusted CAs into the container and pointing the `NODE_EXTRA_CA_CERTS` environment variable to it. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 2a9e0414..18068bd6 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -312,6 +312,8 @@ devture_systemd_service_manager_services_list_auto: | ([{'name': 'matrix-synapse-admin.service', 'priority': 4000, 'groups': ['matrix', 'synapse-admin']}] if matrix_synapse_admin_enabled else []) + ([{'name': 'matrix-synapse-reverse-proxy-companion.service', 'priority': 1500, 'groups': ['matrix', 'homeservers', 'synapse', 'reverse-proxies']}] if matrix_synapse_reverse_proxy_companion_enabled else []) + + + ([{'name': 'matrix-user-verification-service.service', 'priority': 800, 'groups': ['matrix', 'matrix-user-verification-service']}] if matrix_user_verification_service_enabled else []) }} ######################################################################## @@ -1945,6 +1947,11 @@ matrix_jitsi_web_stun_servers: | matrix_jitsi_etherpad_enabled: "{{ matrix_etherpad_enabled }}" matrix_jitsi_etherpad_base: "{{ matrix_etherpad_base_url if matrix_etherpad_enabled else 'https://scalar.vector.im/etherpad' }}" +# Allow verification using JWT and matrix-UVS +matrix_jitsi_prosody_auth_matrix_uvs_auth_token: "{{ matrix_user_verification_service_uvs_auth_token }}" + +matrix_jitsi_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" + ###################################################################### # # /matrix-jitsi @@ -2090,7 +2097,10 @@ matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" matrix_nginx_proxy_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" + matrix_nginx_proxy_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" +matrix_nginx_proxy_proxy_jitsi_manage_wellknown: "{{ matrix_jitsi_require_well_known }}" + matrix_nginx_proxy_proxy_grafana_enabled: "{{ matrix_grafana_enabled }}" matrix_nginx_proxy_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" matrix_nginx_proxy_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" @@ -2732,7 +2742,7 @@ matrix_synapse_tls_federation_listener_enabled: false matrix_synapse_tls_certificate_path: ~ matrix_synapse_tls_private_key_path: ~ -matrix_synapse_federation_port_openid_resource_required: "{{ not matrix_synapse_federation_enabled and (matrix_dimension_enabled or matrix_ma1sd_enabled) }}" +matrix_synapse_federation_port_openid_resource_required: "{{ not matrix_synapse_federation_enabled and (matrix_dimension_enabled or matrix_ma1sd_enabled or matrix_user_verification_service_enabled) }}" # If someone instals Prometheus via the playbook, they most likely wish to monitor Synapse. matrix_synapse_metrics_enabled: "{{ matrix_prometheus_enabled }}" @@ -3166,3 +3176,52 @@ matrix_user_creator_users_auto: | # /matrix-user-creator # ###################################################################### + + +###################################################################### +# +# matrix-user-verification-service +# +###################################################################### + +## FIXME: Needs to be updated when there is a proper release by upstream. +matrix_user_verification_service_docker_image: "{{ matrix_user_verification_service_docker_image_name_prefix }}matrixdotorg/matrix-user-verification-service@sha256:d2aabc984dd69d258c91900c36928972d7aaef19d776caa3cd6a0fbc0e307270" + +# enable if jitsi is managed by this playbook and requires JWT auth +matrix_user_verification_service_enabled: "{{ (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') }}" +matrix_user_verification_service_systemd_required_services_list: | + {{ + ['docker.service'] + + + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + }} + +# If Jitsi is managed by this playbook we can use the docker network - no need to expose a port. +# If Jitsi is not managed by this playbook, or you otherwise have a need for it, you can expose +# matrix-user-verfification-services's client-server port to the local host. +# Note: If grafana is also enabled, the exposed port is changed to 3003. +matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') else '127.0.0.1:' + ('3003' if matrix_nginx_proxy_proxy_grafana_enabled else '3000') }}" + +# URL exposed in the docker network +matrix_user_verification_service_container_url: "http://{{ matrix_user_verification_service_container_name }}:3000" + +# Set the homeserver URL to the container name if synapse is managed by this collection +matrix_user_verification_service_uvs_homeserver_url: "{{ matrix_homeserver_container_url if matrix_synapse_enabled }}" +# If synapse is managed by this collection, we will connect via docker network, which is a private ip. +# Therefore we need to disable IP checks +matrix_user_verification_service_uvs_disable_ip_blacklist: "{{'true' if matrix_synapse_enabled else 'false'}}" + +matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'uvs.auth.token', rounds=655555) | to_uuid }}" + +# Pin UVS to only check openId Tokens for the matrix_server_name configured by this collection. +# This is not the homeserverURL, but rather the domain in the matrix "user ID" +matrix_user_verification_service_uvs_openid_verify_server_name: "{{ matrix_domain }}" +matrix_user_verification_service_uvs_log_level: warning + +###################################################################### +# +# /matrix-user-verification-service +# +###################################################################### diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 088ea640..a74b982d 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -94,6 +94,7 @@ - custom/matrix-client-hydrogen - custom/matrix-client-cinny - custom/matrix-jitsi + - custom/matrix-user-verification-service - custom/matrix-ldap-registration-proxy - custom/matrix-ma1sd - custom/matrix-dimension diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index ce4d19cc..1f8502c8 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -13,14 +13,16 @@ matrix_jitsi_enable_jaas_components: false matrix_jitsi_enable_p2p: true matrix_jitsi_enable_av_moderation: true matrix_jitsi_enable_breakout_rooms: true +matrix_jitsi_disable_gravatar: true -# Authentication type, must be one of internal, jwt or ldap. -# Currently only internal and ldap mechanisms are supported by this playbook. +# Authentication type, must be one of internal, jwt, matrix or ldap. +# Currently, only internal, matrix and ldap mechanisms are supported by this playbook. +# matrix auth verifies against matrix openID, and requires a user-verification-service to run. matrix_jitsi_auth_type: internal # A list of Jitsi (Prosody) accounts to create using the internal authentication mechanism. # -# Accounts added here and subsquently removed will not be automatically removed +# Accounts added here and subsequently removed will not be automatically removed # from the Prosody server until user account cleaning is integrated into the playbook. # # Example: @@ -49,6 +51,23 @@ matrix_jitsi_ldap_tls_cacert_file: "/etc/ssl/certs/ca-certificates.crt" matrix_jitsi_ldap_tls_cacert_dir: "/etc/ssl/certs" matrix_jitsi_ldap_start_tls: false +# Auth type: matrix +matrix_jitsi_prosody_auth_matrix_user_verification_repo_location: "https://github.com/matrix-org/prosody-mod-auth-matrix-user-verification" +matrix_jitsi_prosody_auth_matrix_user_verification_repo_target: "{{ matrix_jitsi_prosody_ext_path }}/prosody_auth_matrix_user_verification" +matrix_jitsi_prosody_auth_matrix_user_verification_repo_version: "2839499cb03894d8cfc3e5b2219441427cb133d8" # v1.8.0 +matrix_jitsi_prosody_auth_matrix_uvs_sync_power_levels: true +matrix_jitsi_prosody_auth_matrix_uvs_location: "{{ matrix_user_verification_service_container_url }}" +# Should match domain, see https://github.com/vector-im/element-web/pull/15114/commits/0410a6b3be82a41457275e4d1ce879dea146e092 +matrix_jitsi_prosody_auth_matrix_jwt_app_id: "{{ matrix_server_fqn_jitsi }}" +matrix_jitsi_prosody_auth_matrix_files: + - path: "mod_auth_matrix_user_verification.lua" + when: true + - path: "mod_matrix_power_sync.lua" + when: "{{ matrix_jitsi_prosody_auth_matrix_uvs_sync_power_levels }}" + +# Plugged in group_vars +#matrix_jitsi_prosody_auth_matrix_uvs_auth_token: + matrix_jitsi_timezone: UTC matrix_jitsi_xmpp_domain: meet.jitsi @@ -180,6 +199,17 @@ matrix_jitsi_prosody_docker_image_force_pull: "{{ matrix_jitsi_prosody_docker_im matrix_jitsi_prosody_base_path: "{{ matrix_base_data_path }}/jitsi/prosody" matrix_jitsi_prosody_config_path: "{{ matrix_jitsi_prosody_base_path }}/config" matrix_jitsi_prosody_plugins_path: "{{ matrix_jitsi_prosody_base_path }}/prosody-plugins-custom" +matrix_jitsi_prosody_ext_path: "{{ matrix_jitsi_prosody_base_path }}/ext" + +# well known is currently only needed for auth type "matrix" +matrix_jitsi_require_well_known: "{{ matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix' }}" +matrix_jitsi_wellknown_element_jitsi_json: '{"auth": "openidtoken-jwt"}' + +# +matrix_jitsi_muc_modules: | + {{ + (['matrix_power_sync'] if matrix_jitsi_prosody_auth_matrix_uvs_sync_power_levels | bool else []) + }} # A list of extra arguments to pass to the container matrix_jitsi_prosody_container_extra_arguments: [] diff --git a/roles/custom/matrix-jitsi/tasks/main.yml b/roles/custom/matrix-jitsi/tasks/main.yml index d7dc6623..b35c2c34 100644 --- a/roles/custom/matrix-jitsi/tasks/main.yml +++ b/roles/custom/matrix-jitsi/tasks/main.yml @@ -67,3 +67,9 @@ - setup-all - setup-jitsi - setup-additional-jitsi-jvb + +- block: + - when: matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix' + ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_matrix_auth.yml" + tags: + - self-check diff --git a/roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml b/roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml new file mode 100644 index 00000000..68a28ef7 --- /dev/null +++ b/roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml @@ -0,0 +1,62 @@ +--- + +- ansible.builtin.set_fact: + matrix_jitsi_prosody_self_check_uvs_health_url: "{{ matrix_jitsi_prosody_auth_matrix_uvs_location }}/health" + matrix_jitsi_element_jitsi_well_known_url: "{{ matrix_jitsi_web_public_url }}/.well-known/element/jitsi" + +- name: Check if jitsi serves the .well-known/element/jitsi + ansible.builtin.uri: + url: "{{ matrix_jitsi_element_jitsi_well_known_url }}" + follow_redirects: none + return_content: true + validate_certs: "{{ matrix_jitsi_self_check_validate_certificates }}" + headers: + Origin: example.com + check_mode: false + register: result_well_known_jitsi_element_jitsi + ignore_errors: true + +- name: Fail if .well-known not working + ansible.builtin.fail: + msg: | + Failed checking that the Jitsi well-known file for Element auth is configured at `{{ matrix_jitsi_element_jitsi_well_known_url }}` + Full error: {{ result_well_known_jitsi_element_jitsi }} + when: "result_well_known_jitsi_element_jitsi.failed" + +- name: Parse JSON for well-known payload at the matrix hostname + ansible.builtin.set_fact: + well_known_matrix_payload: "{{ result_well_known_jitsi_element_jitsi.content | from_json }}" + +- name: Fail if .well-known not CORS-aware + ansible.builtin.fail: + msg: "The well-known file on `{{ matrix_jitsi_element_jitsi_well_known_url }}` is not CORS-aware. The file needs to be served with an Access-Control-Allow-Origin header set." + when: "'access_control_allow_origin' not in result_well_known_jitsi_element_jitsi" + +- name: Report working .well-known + ansible.builtin.debug: + msg: "well-known is configured correctly at `{{ matrix_jitsi_element_jitsi_well_known_url }}`" + +- name: Check if we can reach the user verification service and if it's healthy + ansible.builtin.command: + argv: + - "docker" + - "exec" + - "matrix-jitsi-prosody" + - "wget" + - "-O" + - "-" + - "--quiet" + - "{{ matrix_jitsi_prosody_self_check_uvs_health_url | quote }}" + register: matrix_jitsi_prosody_self_check_uvs_result + ignore_errors: true + +- name: Fail if user verification service is not (reachable and healthy) + ansible.builtin.fail: + msg: | + Failed checking user verification service is up (checked endpoint: `{{ matrix_jitsi_prosody_self_check_uvs_health_url }}`). + Full error: {{ matrix_jitsi_prosody_self_check_uvs_result }} + when: "matrix_jitsi_prosody_self_check_uvs_result.failed" + +- name: Report healthy user verification service + ansible.builtin.debug: + msg: "User verification service is working (checked endpoint: `{{ matrix_jitsi_prosody_self_check_uvs_health_url }}`)" diff --git a/roles/custom/matrix-jitsi/tasks/setup_jitsi_prosody_install.yml b/roles/custom/matrix-jitsi/tasks/setup_jitsi_prosody_install.yml index 78581166..f5beab2b 100644 --- a/roles/custom/matrix-jitsi/tasks/setup_jitsi_prosody_install.yml +++ b/roles/custom/matrix-jitsi/tasks/setup_jitsi_prosody_install.yml @@ -11,6 +11,7 @@ - {path: "{{ matrix_jitsi_prosody_base_path }}", when: true} - {path: "{{ matrix_jitsi_prosody_config_path }}", when: true} - {path: "{{ matrix_jitsi_prosody_plugins_path }}", when: true} + - {path: "{{ matrix_jitsi_prosody_ext_path }}", when: true} when: item.when | bool - name: Ensure jitsi-prosody Docker image is pulled @@ -32,6 +33,43 @@ group: "{{ matrix_user_groupname }}" mode: 0640 +# Configure matrix authentication. +- name: Install user verification plugin + ansible.builtin.include_tasks: + file: "{{ role_path }}/tasks/util/setup_jitsi_auth_uvs_install.yml" + when: matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == "matrix" + +- name: Manage Jitsi .well-known + when: matrix_jitsi_require_well_known | bool + block: + - name: Ensure .well-known directories exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0775 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - {path: "{{ matrix_static_files_base_path }}/.well-known/element", when: "{{ matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix' }}"} + when: item.when | bool + + # Create .well-known/element/jitsi in the static file directory for nginx-proxy. + - name: Ensure Jitsi /.well-known/element/jitsi configured + ansible.builtin.copy: + content: "{{ matrix_jitsi_wellknown_element_jitsi_json }}" + dest: "{{ matrix_static_files_base_path }}/.well-known/element/jitsi" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == "matrix" +# END Block + +# Remove matrix authentication if disabled +- name: Ensure user verification plugin is not present if matrix auth is disabled + ansible.builtin.include_tasks: + file: "{{ role_path }}/tasks/util/setup_jitsi_auth_uvs_uninstall.yml" + when: (not matrix_jitsi_enable_auth | bool) or (matrix_jitsi_auth_type != "matrix") + - name: Ensure matrix-jitsi-prosody.service file is installed ansible.builtin.template: src: "{{ role_path }}/templates/prosody/matrix-jitsi-prosody.service.j2" @@ -39,7 +77,9 @@ mode: 0644 register: matrix_jitsi_prosody_systemd_service_result -- name: Ensure authentication is properly configured +# Tasks that require a running prosody container are called in this file. +- name: Run prosody related tasks, that require a running container. ansible.builtin.include_tasks: - file: "{{ role_path }}/tasks/util/setup_jitsi_auth.yml" - when: matrix_jitsi_enable_auth | bool + file: "{{ role_path }}/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml" + when: + - matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == "internal" diff --git a/roles/custom/matrix-jitsi/tasks/util/prosody_post_setup_hooks/setup_jitsi_auth_internal.yml b/roles/custom/matrix-jitsi/tasks/util/prosody_post_setup_hooks/setup_jitsi_auth_internal.yml new file mode 100644 index 00000000..ac9bda49 --- /dev/null +++ b/roles/custom/matrix-jitsi/tasks/util/prosody_post_setup_hooks/setup_jitsi_auth_internal.yml @@ -0,0 +1,17 @@ +--- +# +# Tasks related to configuring Jitsi internal authentication on a running prosody instance. +# + +- name: Ensure Jitsi internal authentication users are configured + ansible.builtin.shell: "{{ devture_systemd_docker_base_host_command_docker }} exec matrix-jitsi-prosody prosodyctl --config /config/prosody.cfg.lua register {{ item.username | quote }} meet.jitsi {{ item.password | quote }}" + with_items: "{{ matrix_jitsi_prosody_auth_internal_accounts }}" + when: + - matrix_jitsi_prosody_auth_internal_accounts|length > 0 + register: matrix_jitsi_user_configuration_result + changed_when: matrix_jitsi_user_configuration_result.rc == 0 + no_log: true + +# +# Tasks related to clean up after configuring internal authentication. +# diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth.yml deleted file mode 100644 index 60a49b42..00000000 --- a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth.yml +++ /dev/null @@ -1,42 +0,0 @@ ---- -# -# Start Necessary Services -# - -- name: Ensure matrix-jitsi-prosody container is running - ansible.builtin.systemd: - state: started - name: matrix-jitsi-prosody - register: matrix_jitsi_prosody_start_result - - -# -# Tasks related to configuring Jitsi internal authentication -# - -- name: Ensure Jitsi internal authentication users are configured - ansible.builtin.shell: "{{ devture_systemd_docker_base_host_command_docker }} exec matrix-jitsi-prosody prosodyctl --config /config/prosody.cfg.lua register {{ item.username | quote }} meet.jitsi {{ item.password | quote }}" - with_items: "{{ matrix_jitsi_prosody_auth_internal_accounts }}" - when: - - matrix_jitsi_auth_type == "internal" - - matrix_jitsi_prosody_auth_internal_accounts|length > 0 - register: matrix_jitsi_user_configuration_result - changed_when: matrix_jitsi_user_configuration_result.rc == 0 - no_log: true - -# -# Tasks related to configuring other Jitsi authentication mechanisms -# - -# -# Tasks related to cleaning after Jitsi authentication configuration -# - -# -# Stop Necessary Services -# -- name: Ensure matrix-jitsi-prosody container is stopped if necessary - ansible.builtin.systemd: - state: stopped - name: matrix-jitsi-prosody - when: matrix_jitsi_prosody_start_result.changed | bool diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml new file mode 100644 index 00000000..36f33425 --- /dev/null +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml @@ -0,0 +1,13 @@ +- name: Checkout Prosody Auth Matrix User Verification Plugin Repo + ansible.builtin.git: + repo: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_location }}" + dest: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_target }}" + version: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_version }}" + +- name: Install Prosody Auth Matrix User Verification Plugin + ansible.builtin.copy: + remote_src: yes + src: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_target }}/{{ item.path }}" + dest: "{{ matrix_jitsi_prosody_plugins_path }}/{{ item.path }}" + with_items: "{{ matrix_jitsi_prosody_auth_matrix_files }}" + when: item.when | bool diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml new file mode 100644 index 00000000..2ba793ec --- /dev/null +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml @@ -0,0 +1,26 @@ +- name: Remove all files regarding prosody mod auth_matrix_user_verification and .well-known/element/jitsi + ansible.builtin.file: + path: "{{ item }}" + state: absent + with_flattened: + - "{{ matrix_static_files_base_path }}/.well-known/element/jitsi" + - "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_target }}" + - "{{ matrix_jitsi_prosody_auth_matrix_files | map(attribute='path') | map('regex_replace', '^', matrix_jitsi_prosody_plugins_path+'/') | list }}" + register: matrix_jitsi_prosody_auth_matrix_user_verification_uninstalled + +- name: Remove .well-known/element directory if empty + ansible.builtin.command: + argv: + - rmdir + - "{{ matrix_static_files_base_path }}/.well-known/element" + removes: "{{matrix_static_files_base_path}}/.well-known/element" + ignore_errors: yes + +- when: matrix_jitsi_prosody_auth_matrix_user_verification_uninstalled.changed + block: + - name: Populate service facts + ansible.builtin.service_facts: + + - name: Ensure prosody is restarted later on if currently running + set_fact: + matrix_jitsi_prosody_require_restart: "{{ true if ansible_facts.services['matrix-jitsi-prosody.service']['state'] == 'running' else false }}" diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml new file mode 100644 index 00000000..69c18ab3 --- /dev/null +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml @@ -0,0 +1,49 @@ +--- +##### +# +# This tasks file starts and stops (if state before was stopped) a prosody container during setup to run commands, +# that require a running prosody container. +# The task is called in ../setup_jitsi_prosody_install.yml. +# +# Important: The task is called conditionally, as to only start if really needed. +# So if you add or change anything - remember to also change the 'when' in: ../setup_jitsi_prosody_install.yml +# +##### + +# +# Start Necessary Services +# + +- name: Ensure matrix-jitsi-prosody container is running + ansible.builtin.systemd: + state: "{{ 'restarted' if matrix_jitsi_prosody_require_restart | d(false) | bool else 'started' }}" + name: matrix-jitsi-prosody + register: matrix_jitsi_prosody_start_result + +# If the flag was set, we can safely disable now. +- name: Disable require restart flag + set_fact: + matrix_jitsi_prosody_require_restart: false + +# +# Tasks related to configuring Jitsi internal authentication +# + +- name: Ensure internal authentication is properly configured + ansible.builtin.include_tasks: + file: "{{ role_path }}/tasks/util/prosody_post_setup_hooks/setup_jitsi_auth_internal.yml" + when: matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == "internal" + +# +# Tasks related to ... +# + + +# +# Stop Necessary Services +# +- name: Ensure matrix-jitsi-prosody container is stopped if necessary + ansible.builtin.systemd: + state: stopped + name: matrix-jitsi-prosody + when: matrix_jitsi_prosody_start_result.changed | bool diff --git a/roles/custom/matrix-jitsi/tasks/validate_config.yml b/roles/custom/matrix-jitsi/tasks/validate_config.yml index 258b4864..5975a605 100644 --- a/roles/custom/matrix-jitsi/tasks/validate_config.yml +++ b/roles/custom/matrix-jitsi/tasks/validate_config.yml @@ -25,16 +25,25 @@ - "matrix_jitsi_jvb_auth_password" -- name: Fail if a Jitsi internal authentication account is not defined +- name: Fail if authentication is enabled, but not properly configured. ansible.builtin.fail: msg: >- + You have enabled authentication, but the configured auth type is missing required configuration. + + Auth type 'internal': At least one Jitsi user needs to be defined in `matrix_jitsi_prosody_auth_internal_accounts` when using internal authentication. If you're setting up Jitsi for the first time, you may have missed a step. Refer to our setup instructions (docs/configuring-playbook-jitsi.md). + + Auth type 'matrix': + If you want to enable matrix_user_verification in jitsi, + please provide an auth token for the user verification service (uvs) using `matrix_jitsi_prosody_auth_matrix_uvs_auth_token`. + If the user-verfication-service is also managed by this playbook the token is derived from `matrix_homeserver_generic_secret_key` in the group vars. when: - matrix_jitsi_enable_auth | bool - - matrix_jitsi_auth_type == 'internal' - - matrix_jitsi_prosody_auth_internal_accounts|length == 0 + - ((matrix_jitsi_auth_type == 'internal' and matrix_jitsi_prosody_auth_internal_accounts|length == 0) + or (matrix_jitsi_auth_type == 'matrix' and matrix_jitsi_prosody_auth_matrix_uvs_auth_token|length == 0)) + - name: (Deprecation) Catch and report renamed settings diff --git a/roles/custom/matrix-jitsi/templates/prosody/env.j2 b/roles/custom/matrix-jitsi/templates/prosody/env.j2 index b0ebbefc..541727b8 100644 --- a/roles/custom/matrix-jitsi/templates/prosody/env.j2 +++ b/roles/custom/matrix-jitsi/templates/prosody/env.j2 @@ -41,6 +41,12 @@ LDAP_TLS_CACERT_DIR={{ matrix_jitsi_ldap_tls_cacert_dir }} LDAP_START_TLS={{ 1 if matrix_jitsi_ldap_start_tls else 0 }} LDAP_URL={{ matrix_jitsi_ldap_url }} LDAP_USE_TLS={{ 1 if matrix_jitsi_ldap_use_tls else 0 }} +MATRIX_UVS_ISSUER={{ matrix_jitsi_prosody_auth_matrix_jwt_app_id }} +MATRIX_UVS_URL={{ matrix_jitsi_prosody_auth_matrix_uvs_location }} +{% if matrix_jitsi_prosody_auth_matrix_uvs_auth_token is defined %} +MATRIX_UVS_AUTH_TOKEN={{ matrix_jitsi_prosody_auth_matrix_uvs_auth_token }} +{% endif %} +MATRIX_UVS_SYNC_POWER_LEVELS={{ 'true' if matrix_jitsi_prosody_auth_matrix_uvs_sync_power_levels else 'false' }} PUBLIC_URL={{ matrix_jitsi_web_public_url }} TURN_CREDENTIALS={{ matrix_jitsi_turn_credentials }} TURN_HOST={{ matrix_jitsi_turn_host }} @@ -55,7 +61,7 @@ XMPP_GUEST_DOMAIN={{ matrix_jitsi_xmpp_guest_domain }} XMPP_MUC_DOMAIN={{ matrix_jitsi_xmpp_muc_domain }} XMPP_INTERNAL_MUC_DOMAIN={{ matrix_jitsi_xmpp_internal_muc_domain }} XMPP_MODULES={{ matrix_jitsi_xmpp_modules }} -XMPP_MUC_MODULES= +XMPP_MUC_MODULES={{ matrix_jitsi_muc_modules | join(',') }} XMPP_INTERNAL_MUC_MODULES= XMPP_RECORDER_DOMAIN={{ matrix_jitsi_recorder_domain }} XMPP_CROSS_DOMAIN=true diff --git a/roles/custom/matrix-jitsi/templates/web/custom-config.js.j2 b/roles/custom/matrix-jitsi/templates/web/custom-config.js.j2 index bbe85798..2bde96a8 100644 --- a/roles/custom/matrix-jitsi/templates/web/custom-config.js.j2 +++ b/roles/custom/matrix-jitsi/templates/web/custom-config.js.j2 @@ -11,6 +11,10 @@ config.p2p.stunServers = [ ]; {% endif %} +{% if matrix_jitsi_disable_gravatar %} +config.gravatar = {'disabled': true}; +{% endif %} + {% if matrix_jitsi_etherpad_enabled %} config.etherpad_base = {{ (matrix_jitsi_etherpad_base + '/p/') |to_json }} {% endif %} diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 6fbc019e..5d5329b0 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -203,6 +203,8 @@ matrix_nginx_proxy_proxy_bot_go_neb_hostname: "{{ matrix_server_fqn_bot_go_neb } # Controls whether proxying the jitsi domain should be done. matrix_nginx_proxy_proxy_jitsi_enabled: false matrix_nginx_proxy_proxy_jitsi_hostname: "{{ matrix_server_fqn_jitsi }}" +matrix_nginx_proxy_proxy_jitsi_manage_wellknown: false +matrix_nginx_proxy_proxy_jitsi_well_known_configuration_blocks: [] # Controls whether proxying the grafana domain should be done. matrix_nginx_proxy_proxy_grafana_enabled: false diff --git a/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-jitsi.conf.j2 b/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-jitsi.conf.j2 index 4d5a4ce7..39dedfc8 100644 --- a/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-jitsi.conf.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-jitsi.conf.j2 @@ -18,6 +18,15 @@ {{- configuration_block }} {% endfor %} + {% if matrix_nginx_proxy_proxy_jitsi_manage_wellknown %} + location /.well-known { + root {{ matrix_static_files_base_path }}; + expires 4h; + default_type application/json; + add_header Access-Control-Allow-Origin *; + } + {% endif %} + location / { {% if matrix_nginx_proxy_enabled %} {# Use the embedded DNS resolver in Docker containers to discover the service #} @@ -75,7 +84,7 @@ {% if matrix_nginx_proxy_enabled %} resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s; set $backend {{ matrix_jitsi_xmpp_bosh_url_base }}; - proxy_pass $backend/xmpp-websocket; + proxy_pass $backend$request_uri; {% else %} {# Generic configuration for use outside of our container setup #} proxy_pass http://127.0.0.1:5280; diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml new file mode 100644 index 00000000..a86d6235 --- /dev/null +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -0,0 +1,81 @@ +--- +# Set this to the display name for ansible used in Output e.g. fail_msg +matrix_user_verification_service_ansible_name: "Matrix User Verification Service" + +# Enable by default. This is overwritten in provided group vars. +matrix_user_verification_service_enabled: true + +# Fix version tag +matrix_user_verification_service_version: "v2.0.0" + +# Paths +matrix_user_verification_service_base_path: "{{ matrix_base_data_path }}/user-verification-service" +# We need the docker src directory to be named user_verification_service. See: https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/588 +matrix_user_verification_service_docker_src_files_path: "{{ matrix_user_verification_service_base_path }}/docker-src/user-verification-service" +matrix_user_verification_service_config_path: "{{ matrix_user_verification_service_base_path }}/config" +matrix_user_verification_service_config_env_file: "{{ matrix_user_verification_service_config_path }}/.env" + +# Set this to true in order to not use the docker image from docker hub, but rather build locally +matrix_user_verification_service_container_image_self_build: false +matrix_user_verification_service_container_image_self_build_repo: "https://github.com/matrix-org/matrix-user-verification-service.git" +matrix_user_verification_service_container_image_self_build_branch: "{{ matrix_user_verification_service_version }}" + +# Docker +matrix_user_verification_service_docker_image_name_prefix: "{{ 'localhost/' if matrix_user_verification_service_container_image_self_build else matrix_container_global_registry_prefix }}" +matrix_user_verification_service_docker_image: "{{ matrix_user_verification_service_docker_image_name_prefix }}matrixdotorg/matrix-user-verification-service:{{ matrix_user_verification_service_version }}" +matrix_user_verification_service_docker_image_force_pull: "{{ matrix_user_verification_service_docker_image.endswith(':latest') }}" + +matrix_user_verification_service_container_name: "matrix-user-verification-service" +# Normally this would run on port 3000 however that may conflict with grafana. It is thus advised to change this port. +#matrix_user_verification_service_container_http_host_bind_port: +matrix_user_verification_service_container_extra_arguments: [] +# Systemd +matrix_user_verification_service_systemd_required_services_list: [] +matrix_user_verification_service_systemd_wanted_services_list: [] +matrix_user_verification_service_systemd_service_basename: "matrix-user-verification-service" +matrix_user_verification_service_systemd_service_name: "{{ matrix_user_verification_service_systemd_service_basename }}.service" + +# Matrix User Verification Service Configuration +## REQUIRED + +# Homeserver client API admin token (synapse only)- Required for the service to verify room membership +# matrix_user_verification_service_uvs_access_token: + +# homeserver client api url +# matrix_user_verification_service_uvs_homeserver_url: "" +# disable check for non private ip range of homeserver. e.g. set to `true` if your homeserver domain resolves to a private ip. +matrix_user_verification_service_uvs_disable_ip_blacklist: false + +## OPTIONAL + +# Auth token to protect the API +# If this is set any calls to the provided API endpoints +# need have the header "Authorization: Bearer changeme". +# matrix_user_verification_service_uvs_auth_token: changeme + +# Matrix server name to verify OpenID tokens against. See below section. +# Defaults to empty value which means verification is made against +# whatever Matrix server name passed in with the token +# matrix_user_verification_service_uvs_openid_verify_server_name: matrix.org + +# Log level, defaults to 'info' +# See choices here: https://github.com/winstonjs/winston#logging-levels +# matrix_user_verification_service_uvs_log_level: info + + +###################################################################### +##### ##### +##### Variables used in this role which are not set by this role ##### +##### ##### +###################################################################### + +# matrix_user_username +# matrix_user_groupname +# matrix_user_uid +# matrix_user_gid +# matrix_container_global_registry_prefix +# matrix_docker_network +# devture_systemd_docker_base_systemd_path +# devture_systemd_docker_base_systemd_unit_home_path +# devture_systemd_docker_base_host_command_sh +# devture_systemd_docker_base_host_command_docker diff --git a/roles/custom/matrix-user-verification-service/handlers/main.yml b/roles/custom/matrix-user-verification-service/handlers/main.yml new file mode 100644 index 00000000..a1ad1f31 --- /dev/null +++ b/roles/custom/matrix-user-verification-service/handlers/main.yml @@ -0,0 +1,6 @@ +--- + +- name: Ensure systemd reloaded after matrix-user-verification-service.service installation + service: + daemon_reload: yes + listen: "reload matrix-user-verification-service" diff --git a/roles/custom/matrix-user-verification-service/tasks/main.yml b/roles/custom/matrix-user-verification-service/tasks/main.yml new file mode 100644 index 00000000..0f51d6cc --- /dev/null +++ b/roles/custom/matrix-user-verification-service/tasks/main.yml @@ -0,0 +1,24 @@ +--- + +- name: verify all necessary variables are present + assert: + that: + - matrix_user_verification_service_uvs_access_token is defined and matrix_user_verification_service_uvs_access_token|length + - matrix_user_verification_service_uvs_homeserver_url is defined and matrix_user_verification_service_uvs_homeserver_url|length + fail_msg: "Missing variable in {{ matrix_user_verification_service_ansible_name }} role" + +- block: + - when: run_setup | bool and matrix_user_verification_service_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + tags: + - setup-all + - setup-user-verification-service + - install-all + - install-user-verification-service + +- block: + - when: run_setup | bool and not matrix_user_verification_service_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + tags: + - setup-all + - setup-user-verification-service diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml new file mode 100644 index 00000000..5d1d2845 --- /dev/null +++ b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml @@ -0,0 +1,42 @@ +--- + +- name: "Ensure Matrix User Verification Service paths exist" + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - { path: "{{ matrix_user_verification_service_config_path }}", when: true } + - { path: "{{ matrix_user_verification_service_docker_src_files_path }}", when: "{{ matrix_user_verification_service_container_image_self_build }}" } + when: item.when | bool + +- name: Ensure Matrix User Verification Service image is pulled + community.docker.docker_image: + name: "{{ matrix_user_verification_service_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_user_verification_service_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_user_verification_service_docker_image_force_pull }}" + when: "not matrix_user_verification_service_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +#- block: +# TODO +# when: "matrix_user_verification_service_container_image_self_build|bool" + +- name: write env file + ansible.builtin.template: + src: "{{ role_path }}/templates/.env.j2" + dest: "{{ matrix_user_verification_service_config_env_file }}" + mode: 0644 + +- name: Ensure matrix-user-verification-service.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-user-verification-service.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" + mode: 0644 + notify: "reload matrix-user-verification-service" diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml new file mode 100644 index 00000000..2a14d96a --- /dev/null +++ b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml @@ -0,0 +1,35 @@ +--- + +- name: Check existence of matrix-user-verification-service service + stat: + path: "{{ matrix_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" + register: matrix_user_verification_service_service_stat + +- name: Ensure matrix-user-verification-service is stopped + service: + name: "{{ matrix_user_verification_service_systemd_service_basename }}" + state: stopped + daemon_reload: yes + register: stopping_result + when: "matrix_user_verification_service_service_stat.stat.exists|bool" + +- name: Ensure matrix-user-verification-service.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" + state: absent + when: "matrix_user_verification_service_service_stat.stat.exists|bool" + +- name: Ensure systemd reloaded after matrix-user-verification-service.service removal + service: + daemon_reload: yes + when: "matrix_user_verification_service_service_stat.stat.exists|bool" + +- name: Ensure Matrix user-verification-service paths don't exist + file: + path: "{{ matrix_user_verification_service_base_path }}" + state: absent + +- name: Ensure user-verification-service Docker image doesn't exist + docker_image: + name: "{{ matrix_user_verification_service_docker_image }}" + state: absent diff --git a/roles/custom/matrix-user-verification-service/templates/.env.j2 b/roles/custom/matrix-user-verification-service/templates/.env.j2 new file mode 100644 index 00000000..b2f2aaab --- /dev/null +++ b/roles/custom/matrix-user-verification-service/templates/.env.j2 @@ -0,0 +1,14 @@ +UVS_ACCESS_TOKEN={{ matrix_user_verification_service_uvs_access_token }} +UVS_HOMESERVER_URL={{ matrix_user_verification_service_uvs_homeserver_url }} +UVS_DISABLE_IP_BLACKLIST={{ matrix_user_verification_service_uvs_disable_ip_blacklist }} + +{% if matrix_user_verification_service_uvs_auth_token is defined and matrix_user_verification_service_uvs_auth_token|length %} + UVS_AUTH_TOKEN={{ matrix_user_verification_service_uvs_auth_token }} +{% endif %} +{% if matrix_user_verification_service_uvs_openid_verify_server_name is defined and matrix_user_verification_service_uvs_openid_verify_server_name|length %} + UVS_OPENID_VERIFY_SERVER_NAME={{ matrix_user_verification_service_uvs_openid_verify_server_name }} +{% endif %} +{% if matrix_user_verification_service_uvs_log_level is defined and matrix_user_verification_service_uvs_log_level|length %} + UVS_LOG_LEVEL={{ matrix_user_verification_service_uvs_log_level }} +{% endif %} + diff --git a/roles/custom/matrix-user-verification-service/templates/systemd/matrix-user-verification-service.service.j2 b/roles/custom/matrix-user-verification-service/templates/systemd/matrix-user-verification-service.service.j2 new file mode 100644 index 00000000..eb24b128 --- /dev/null +++ b/roles/custom/matrix-user-verification-service/templates/systemd/matrix-user-verification-service.service.j2 @@ -0,0 +1,42 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description={{ matrix_user_verification_service_ansible_name }} +{% for service in matrix_user_verification_service_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_user_verification_service_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill {{ matrix_user_verification_service_container_name }} 2>/dev/null' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_user_verification_service_container_name }} 2>/dev/null' + + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_user_verification_service_container_name }}\ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --network={{ matrix_docker_network }} \ + {% if matrix_user_verification_service_container_http_host_bind_port %} + -p {{ matrix_user_verification_service_container_http_host_bind_port }}:3000 \ + {% endif %} + --mount type=bind,src={{ matrix_user_verification_service_config_env_file }},dst=/app/.env,ro \ + {% for arg in matrix_user_verification_service_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_user_verification_service_docker_image }} + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill {{ matrix_user_verification_service_container_name }} 2>/dev/null' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_user_verification_service_container_name }} 2>/dev/null' +Restart=always +RestartSec=30 +SyslogIdentifier={{ matrix_user_verification_service_systemd_service_basename }} + +[Install] +WantedBy=multi-user.target From 18d8d41e1454680b404e4aa92609d0cf82f1074d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian-Samuel=20Geb=C3=BChr?= Date: Sat, 21 Jan 2023 20:54:12 +0100 Subject: [PATCH 002/382] Fix client js and css download (load compiled version) --- .../matrix-cactus-comments/defaults/main.yml | 6 ++-- .../tasks/setup_install.yml | 31 +++---------------- 2 files changed, 9 insertions(+), 28 deletions(-) diff --git a/roles/custom/matrix-cactus-comments/defaults/main.yml b/roles/custom/matrix-cactus-comments/defaults/main.yml index 224a5348..7b12b9f1 100644 --- a/roles/custom/matrix-cactus-comments/defaults/main.yml +++ b/roles/custom/matrix-cactus-comments/defaults/main.yml @@ -31,10 +31,12 @@ matrix_cactus_comments_version: 0.9.0 matrix_cactus_comments_docker_image: "{{ matrix_container_global_registry_prefix }}cactuscomments/cactus-appservice:{{ matrix_cactus_comments_version }}" matrix_cactus_comments_docker_image_force_pull: "{{ matrix_cactus_comments_docker_image.endswith(':latest') }}" -# matrix_cactus_comments_client_version specifies the version of the cactus-client release to use. +# The released&compiled client ist uploaded to URL that changes per commit. Therefore it is necessary to check the +# available versions and adjust the url manually. # For available versions, see: https://gitlab.com/cactus-comments/cactus-client/-/releases # Also see: `matrix_cactus_comments_client_local_dir` -matrix_cactus_comments_client_version: "0.13.0" +matrix_cactus_comments_webclient_js_url: "https://gateway.pinata.cloud/ipfs/QmSiWN27KZZ1XE32jKwifBnS3nWTUcFGNArKzur2nmDgoL/v0.13.0/cactus.js" +matrix_cactus_comments_webclient_css_url: "https://gateway.pinata.cloud/ipfs/QmSiWN27KZZ1XE32jKwifBnS3nWTUcFGNArKzur2nmDgoL/v0.13.0/style.css" # matrix_cactus_comments_client_local_dir specifies a local directory (on the Ansible controller, not on the remote server) with cactus-client files to use. # This is an alternative to `matrix_cactus_comments_client_version`, to be used when you'd like to diff --git a/roles/custom/matrix-cactus-comments/tasks/setup_install.yml b/roles/custom/matrix-cactus-comments/tasks/setup_install.yml index 8a8bf326..03b97d0a 100644 --- a/roles/custom/matrix-cactus-comments/tasks/setup_install.yml +++ b/roles/custom/matrix-cactus-comments/tasks/setup_install.yml @@ -67,37 +67,16 @@ - when: matrix_cactus_comments_client_local_dir | length == 0 block: - - name: Download client binary to local folder + - name: Download web client js ansible.builtin.get_url: - url: "https://gitlab.com/cactus-comments/cactus-client/-/archive/v{{ matrix_cactus_comments_client_version }}/cactus-client-v{{ matrix_cactus_comments_client_version }}.tar.gz" - dest: "/tmp/cactus-comments-{{ matrix_cactus_comments_client_version }}.tar.gz" - mode: '0644' - register: _download_client - until: _download_client is succeeded - retries: 5 - delay: 2 - check_mode: false - - - name: Unpack client - ansible.builtin.unarchive: - src: "/tmp/cactus-comments-{{ matrix_cactus_comments_client_version }}.tar.gz" - dest: "/tmp/" - remote_src: true - mode: 0600 - check_mode: false - - - name: Propagate client javascript file - ansible.builtin.copy: - src: "/tmp/cactus-client-v{{ matrix_cactus_comments_client_version }}/src/cactus.js" - remote_src: true + url: "{{ matrix_cactus_comments_webclient_js_url }}" dest: "{{ matrix_cactus_comments_client_path }}/cactus.js" mode: "{{ matrix_cactus_comments_client_file_permissions }}" owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" - - name: Propagate client style file - ansible.builtin.copy: - src: "/tmp/cactus-client-v{{ matrix_cactus_comments_client_version }}/src/style.css" - remote_src: true + - name: Download web client css + ansible.builtin.get_url: + url: "{{ matrix_cactus_comments_webclient_css_url }}" dest: "{{ matrix_cactus_comments_client_path }}/style.css" mode: "{{ matrix_cactus_comments_client_file_permissions }}" owner: "{{ matrix_user_username }}" From c4d1347466d793874383abcee3c5bbb35b0be154 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 22 Jan 2023 09:08:47 +0200 Subject: [PATCH 003/382] Fix typos --- roles/custom/matrix-cactus-comments/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-cactus-comments/defaults/main.yml b/roles/custom/matrix-cactus-comments/defaults/main.yml index 7b12b9f1..80f8c15e 100644 --- a/roles/custom/matrix-cactus-comments/defaults/main.yml +++ b/roles/custom/matrix-cactus-comments/defaults/main.yml @@ -31,7 +31,7 @@ matrix_cactus_comments_version: 0.9.0 matrix_cactus_comments_docker_image: "{{ matrix_container_global_registry_prefix }}cactuscomments/cactus-appservice:{{ matrix_cactus_comments_version }}" matrix_cactus_comments_docker_image_force_pull: "{{ matrix_cactus_comments_docker_image.endswith(':latest') }}" -# The released&compiled client ist uploaded to URL that changes per commit. Therefore it is necessary to check the +# The released & compiled client is uploaded to a URL that changes per commit. Therefore it is necessary to check the # available versions and adjust the url manually. # For available versions, see: https://gitlab.com/cactus-comments/cactus-client/-/releases # Also see: `matrix_cactus_comments_client_local_dir` From 47165e890272f0e6763d7c6c0c68d624377313a8 Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Tue, 24 Jan 2023 15:50:06 -0500 Subject: [PATCH 004/382] updated dendrite version --- roles/custom/matrix-dendrite/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 32f262ad..6a495f8f 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -6,7 +6,7 @@ matrix_dendrite_enabled: true matrix_dendrite_docker_image: "{{ matrix_dendrite_docker_image_name_prefix }}matrixdotorg/dendrite-monolith:{{ matrix_dendrite_docker_image_tag }}" matrix_dendrite_docker_image_name_prefix: "docker.io/" -matrix_dendrite_docker_image_tag: "v0.10.8" +matrix_dendrite_docker_image_tag: "v0.11.0" matrix_dendrite_docker_image_force_pull: "{{ matrix_dendrite_docker_image.endswith(':latest') }}" matrix_dendrite_base_path: "{{ matrix_base_data_path }}/dendrite" From 6afd3c59eb617d2781581578048fb403bd42fd32 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Wed, 25 Jan 2023 08:39:47 +0000 Subject: [PATCH 005/382] Update grafana 9.3.2 -> 9.3.4 --- roles/custom/matrix-grafana/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-grafana/defaults/main.yml b/roles/custom/matrix-grafana/defaults/main.yml index 44821b57..9920572d 100644 --- a/roles/custom/matrix-grafana/defaults/main.yml +++ b/roles/custom/matrix-grafana/defaults/main.yml @@ -5,7 +5,7 @@ matrix_grafana_enabled: true -matrix_grafana_version: 9.3.2 +matrix_grafana_version: 9.3.4 matrix_grafana_docker_image: "{{ matrix_container_global_registry_prefix }}grafana/grafana:{{ matrix_grafana_version }}" matrix_grafana_docker_image_force_pull: "{{ matrix_grafana_docker_image.endswith(':latest') }}" From 547b01d618aad45abcc438a21632b9a9892d6f80 Mon Sep 17 00:00:00 2001 From: rhys Date: Wed, 25 Jan 2023 12:02:40 +0000 Subject: [PATCH 006/382] Added option to allow user to set jigasi user and password for AUTH --- roles/custom/matrix-jitsi/defaults/main.yml | 5 +++++ roles/custom/matrix-jitsi/templates/prosody/env.j2 | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index da166cd8..6b0e7e96 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -283,3 +283,8 @@ matrix_jitsi_jvb_container_colibri_ws_host_bind_port: '' # # The setting requires an integer to be set for usage and allows a user to specify the max number of particpants on a conference. matrix_prosody_jitsi_max_participants: '' + +# Allows abailty to set XMPP AUTH user if using JIGASI +matrix_jitsi_jigasi_xmpp_user: '' +# Allows abailty to set XMPP AUTH pasword if using JIGASI +matrix_jitsi_jigasi_xmpp_password: '' \ No newline at end of file diff --git a/roles/custom/matrix-jitsi/templates/prosody/env.j2 b/roles/custom/matrix-jitsi/templates/prosody/env.j2 index b0ebbefc..72b23e3c 100644 --- a/roles/custom/matrix-jitsi/templates/prosody/env.j2 +++ b/roles/custom/matrix-jitsi/templates/prosody/env.j2 @@ -15,8 +15,8 @@ JIBRI_XMPP_PASSWORD={{ matrix_jitsi_jibri_xmpp_password }} JICOFO_AUTH_USER={{ matrix_jitsi_jicofo_auth_user }} JICOFO_AUTH_PASSWORD={{ matrix_jitsi_jicofo_auth_password }} JICOFO_COMPONENT_SECRET -JIGASI_XMPP_USER= -JIGASI_XMPP_PASSWORD= +JIGASI_XMPP_USER={{ matrix_jitsi_jigasi_xmpp_user }} +JIGASI_XMPP_PASSWORD={{ matrix_jitsi_jigasi_xmpp_password }} JVB_AUTH_USER={{ matrix_jitsi_jvb_auth_user }} JVB_AUTH_PASSWORD={{ matrix_jitsi_jvb_auth_password }} JWT_APP_ID From d01de9f33dfbf47c636bd796b3a71a9f08a73390 Mon Sep 17 00:00:00 2001 From: rhys Date: Wed, 25 Jan 2023 12:07:00 +0000 Subject: [PATCH 007/382] Fix lint errors --- roles/custom/matrix-jitsi/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index 6b0e7e96..538d8e4e 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -287,4 +287,4 @@ matrix_prosody_jitsi_max_participants: '' # Allows abailty to set XMPP AUTH user if using JIGASI matrix_jitsi_jigasi_xmpp_user: '' # Allows abailty to set XMPP AUTH pasword if using JIGASI -matrix_jitsi_jigasi_xmpp_password: '' \ No newline at end of file +matrix_jitsi_jigasi_xmpp_password: '' From ad58858a965896a27967d086315d5d20c1d8b30e Mon Sep 17 00:00:00 2001 From: bertybuttface <110790513+bertybuttface@users.noreply.github.com> Date: Thu, 26 Jan 2023 15:09:24 +0000 Subject: [PATCH 008/382] Upgrade matrix-chatgpt (1.2.3 -> 1.3.2) Add support for OPENAI_PRO for ChatGPT pro subscriptions. --- roles/custom/matrix-bot-chatgpt/defaults/main.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index d6119ca6..86d185b7 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_chatgpt_docker_repo: "https://github.com/matrixgpt/matrix-chatgpt-bot matrix_bot_chatgpt_docker_repo_version: "{{ 'latest' if matrix_bot_chatgpt_version == 'latest' else matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_src_files_path: "{{ matrix_base_data_path }}/chatgpt/docker-src" -matrix_bot_chatgpt_version: 1.2.3 +matrix_bot_chatgpt_version: 1.3.2 matrix_bot_chatgpt_docker_image: "{{ matrix_bot_chatgpt_docker_image_name_prefix }}matrixgpt/matrix-chatgpt-bot:{{ matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_chatgpt_container_image_self_build else 'ghcr.io/' }}" matrix_bot_chatgpt_docker_image_force_pull: "{{ matrix_bot_chatgpt_docker_image.endswith(':latest') }}" @@ -31,6 +31,7 @@ matrix_bot_chatgpt_systemd_wanted_services_list: [] matrix_bot_chatgpt_openai_email: '' # OPENAI_EMAIL= matrix_bot_chatgpt_openai_password: '' # OPENAI_PASSWORD= matrix_bot_chatgpt_openai_login_type: 'google' # OPENAI_LOGIN_TYPE="google" +matrix_bot_chatgpt_openai_pro: false # OPENAI_PRO # Matrix Static Settings (required, see notes) # Defaults to "https://matrix.org" From 18c3903defbf28a753d4bfebbbe94725a2510f21 Mon Sep 17 00:00:00 2001 From: bertybuttface <110790513+bertybuttface@users.noreply.github.com> Date: Thu, 26 Jan 2023 15:12:36 +0000 Subject: [PATCH 009/382] Update env.j2 Set OPEN_AI to matrix_bot_chatgpt_openai_pro --- roles/custom/matrix-bot-chatgpt/templates/env.j2 | 1 + 1 file changed, 1 insertion(+) diff --git a/roles/custom/matrix-bot-chatgpt/templates/env.j2 b/roles/custom/matrix-bot-chatgpt/templates/env.j2 index 8837fe3e..bf2dafbf 100644 --- a/roles/custom/matrix-bot-chatgpt/templates/env.j2 +++ b/roles/custom/matrix-bot-chatgpt/templates/env.j2 @@ -4,6 +4,7 @@ MATRIX_ACCESS_TOKEN={{ matrix_bot_chatgpt_matrix_access_token }} OPENAI_EMAIL={{ matrix_bot_chatgpt_openai_email }} OPENAI_PASSWORD={{ matrix_bot_chatgpt_openai_password }} OPENAI_LOGIN_TYPE={{ matrix_bot_chatgpt_openai_login_type }} +OPENAI_PRO={{ matrix_bot_chatgpt_openai_pro }} # With the @ and :DOMAIN, ie @SOMETHING:DOMAIN MATRIX_BOT_USERNAME={{ matrix_bot_chatgpt_matrix_bot_username }} From 4c9f96722f3f6ed56e73cfeecbbeb6fd88ab2faf Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 17:04:04 +0200 Subject: [PATCH 010/382] Add no-multicast-peers to Coturn config by default Part of a security hardening provoked by: https://www.rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/ --- roles/custom/matrix-coturn/defaults/main.yml | 3 +++ roles/custom/matrix-coturn/templates/turnserver.conf.j2 | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 3a125de4..22ad48d6 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -73,6 +73,9 @@ matrix_coturn_denied_peer_ips: [] matrix_coturn_user_quota: null matrix_coturn_total_quota: null +# Controls whether `no-multicast-peers` is added to the configuration +matrix_coturn_no_multicast_peers_enabled: true + # To enable TLS, you need to provide paths to certificates. # Paths defined in `matrix_coturn_tls_cert_path` and `matrix_coturn_tls_key_path` are in-container paths. # Files on the host can be mounted into the container using `matrix_coturn_container_additional_volumes`. diff --git a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 index 1bdf310a..dc8d96ee 100644 --- a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 +++ b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 @@ -39,6 +39,10 @@ user-quota={{ matrix_coturn_user_quota }} total-quota={{ matrix_coturn_total_quota }} {% endif %} +{% if matrix_coturn_no_multicast_peers_enabled %} +no-multicast-peers +{% endif %} + {% for ip_range in matrix_coturn_denied_peer_ips %} denied-peer-ip={{ ip_range }} {% endfor %} From bf23d63f8211135ad38288978d0fbf38d193b5df Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 17:07:36 +0200 Subject: [PATCH 011/382] Add matrix_coturn_additional_configuration --- roles/custom/matrix-coturn/defaults/main.yml | 8 ++++++++ roles/custom/matrix-coturn/templates/turnserver.conf.j2 | 2 ++ 2 files changed, 10 insertions(+) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 22ad48d6..0d5d15dc 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -76,6 +76,14 @@ matrix_coturn_total_quota: null # Controls whether `no-multicast-peers` is added to the configuration matrix_coturn_no_multicast_peers_enabled: true +# Additional configuration to be passed to turnserver.conf +# Example: +# matrix_coturn_additional_configuration: | +# simple-log +# aux-server=1.2.3.4 +# relay-ip=4.3.2.1 +matrix_coturn_additional_configuration: '' + # To enable TLS, you need to provide paths to certificates. # Paths defined in `matrix_coturn_tls_cert_path` and `matrix_coturn_tls_key_path` are in-container paths. # Files on the host can be mounted into the container using `matrix_coturn_container_additional_volumes`. diff --git a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 index dc8d96ee..47fa5b86 100644 --- a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 +++ b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 @@ -49,3 +49,5 @@ denied-peer-ip={{ ip_range }} {% for ip_range in matrix_coturn_allowed_peer_ips %} allowed-peer-ip={{ ip_range }} {% endfor %} + +{{ matrix_coturn_additional_configuration }} From 773cb7d37e018ad462d4229bf58ecbe569c918be Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 17:08:05 +0200 Subject: [PATCH 012/382] Make no-tcp-relay Coturn configuration property configurable --- roles/custom/matrix-coturn/defaults/main.yml | 3 +++ roles/custom/matrix-coturn/templates/turnserver.conf.j2 | 3 +++ 2 files changed, 6 insertions(+) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 0d5d15dc..71c97b25 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -73,6 +73,9 @@ matrix_coturn_denied_peer_ips: [] matrix_coturn_user_quota: null matrix_coturn_total_quota: null +# Controls whether `no-tcp-relay` is added to the configuration +matrix_coturn_no_tcp_relay_enabled: true + # Controls whether `no-multicast-peers` is added to the configuration matrix_coturn_no_multicast_peers_enabled: true diff --git a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 index 47fa5b86..dfc3a87d 100644 --- a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 +++ b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 @@ -30,7 +30,10 @@ no-dtls {% endif %} prod + +{% if matrix_coturn_no_tcp_relay_enabled %} no-tcp-relay +{% endif %} {% if matrix_coturn_user_quota != None %} user-quota={{ matrix_coturn_user_quota }} From bb0faa6bc37be94f443fa9730f9af726f9ba1486 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 17:15:44 +0200 Subject: [PATCH 013/382] Block various private network ranges via denied_peer_ips for Coturn by default Inspired by: https://www.rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/ --- CHANGELOG.md | 11 +++++++ roles/custom/matrix-coturn/defaults/main.yml | 32 +++++++++++++++++++- 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4607148..b32958ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,14 @@ +# 2023-01-26 + +## (Backward Compatibility) Tightening Coturn security can lead to connectivity issues + +**TLDR**: users who run and access their Matrix server on a private network (likely a small minority of users) may experience connectivity issues with our new default Coturn blocklists. They may need to override `matrix_coturn_denied_peer_ips` and remove some IP ranges from it. + +Inspired by [this security article](https://www.rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/), we've decided to make use of Coturn's `denied-peer-ip` functionality to prevent relaying network traffic to certain private IP subnets. This ensures that your Coturn server won't accidentally try to forward traffic to certain services running on your local networks. We run Coturn in a container and in a private container network by default, which should prevent such access anyway, but having additional block layers in place is better. + +If you access your Matrix server from a local network and need Coturn to relay to private IP addresses, you may observe that relaying is now blocked due to our new default `denied-peer-ip` lists (specified in `matrix_coturn_denied_peer_ips`). If you experience such connectivity problems, consider overriding this setting in your `vars.yml` file and removing certain networks from it. + + # 2023-01-21 ## The matrix-prometheus-node-exporter role lives independently now diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 71c97b25..223d0d04 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -69,7 +69,37 @@ matrix_coturn_turn_external_ip_address: '' matrix_coturn_turn_external_ip_addresses: ["{{ matrix_coturn_turn_external_ip_address }}"] matrix_coturn_allowed_peer_ips: [] -matrix_coturn_denied_peer_ips: [] + +# We block loopback interfaces and private networks by default to prevent private resources from being accessible. +# This is especially important when Coturn does not run within a container network (e.g. `matrix_coturn_docker_network: host`). +# +# Learn more: https://www.rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/ +# +# If you're running Coturn for local network peers, you may wish to override these rules. +matrix_coturn_denied_peer_ips: + - 0.0.0.0-0.255.255.255 + - 10.0.0.0-10.255.255.255 + - 100.64.0.0-100.127.255.255 + - 127.0.0.0-127.255.255.255 + - 169.254.0.0-169.254.255.255 + - 172.16.0.0-172.31.255.255 + - 192.0.0.0-192.0.0.255 + - 192.0.2.0-192.0.2.255 + - 192.88.99.0-192.88.99.255 + - 192.168.0.0-192.168.255.255 + - 198.18.0.0-198.19.255.255 + - 198.51.100.0-198.51.100.255 + - 203.0.113.0-203.0.113.255 + - 240.0.0.0-255.255.255.255 + - ::1 + - 64:ff9b::-64:ff9b::ffff:ffff + - ::ffff:0.0.0.0-::ffff:255.255.255.255 + - 100::-100::ffff:ffff:ffff:ffff + - 2001::-2001:1ff:ffff:ffff:ffff:ffff:ffff:ffff + - 2002::-2002:ffff:ffff:ffff:ffff:ffff:ffff:ffff + - fc00::-fdff:ffff:ffff:ffff:ffff:ffff:ffff:ffff + - fe80::-febf:ffff:ffff:ffff:ffff:ffff:ffff:ffff + matrix_coturn_user_quota: null matrix_coturn_total_quota: null From aafa8f019cbc3535cc90eb285e5126ccaf415aa4 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 17:16:20 +0200 Subject: [PATCH 014/382] Allow matrix_coturn_docker_network to be set to 'host' to use host-networking This helps large deployments which need to open up thousands of ports (matrix_coturn_turn_udp_min_port, matrix_coturn_turn_udp_min_port) On a test VM, opening 1k ports takes 17 seconds for Docker to "publish" all of these ports (setting up forwarding rules with the firewall, etc), so service startup and shutdown take a long amount of time. If host-networking is used, there's no need to open any ports at all and startup/shutdown can be quick. --- CHANGELOG.md | 16 ++++++++++++++++ roles/custom/matrix-coturn/defaults/main.yml | 15 +++++++++++---- .../custom/matrix-coturn/tasks/setup_install.yml | 3 ++- .../templates/systemd/matrix-coturn.service.j2 | 2 +- 4 files changed, 30 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b32958ca..030fbbdb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,21 @@ # 2023-01-26 +## Coturn can now use host-networking + +Large Coturn deployments (with a huge range of ports specified via `matrix_coturn_turn_udp_min_port` and `matrix_coturn_turn_udp_max_port`) experience a huge slowdown with how Docker publishes all these ports (setting up firewall forwarding rules), which leads to a very slow Coturn service startup and shutdown. + +Such deployments don't need to run Coturn within a private container network anymore. Coturn can now run with host-networking by using configuration like this: + +```yaml +matrix_coturn_docker_network: host +``` + +With such a configuration, Docker no longer needs to configure thousands of firewall forwarding rules each time Coturn starts and stops. +You may, however, need to allow these ports in your firewall configuration yourself. + +Thanks to us [tightening Coturn security](#backward-compatibility-tightening-coturn-security-can-lead-to-connectivity-issues), running Coturn with host-networking should be safe and not expose neither other services running on the host, nor other services running on the local network. + + ## (Backward Compatibility) Tightening Coturn security can lead to connectivity issues **TLDR**: users who run and access their Matrix server on a private network (likely a small minority of users) may experience connectivity issues with our new default Coturn blocklists. They may need to override `matrix_coturn_denied_peer_ips` and remove some IP ranges from it. diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 223d0d04..9f7ff2ba 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -20,6 +20,13 @@ matrix_coturn_docker_image_force_pull: "{{ matrix_coturn_docker_image.endswith(' # # Setting up deny/allow rules with `matrix_coturn_allowed_peer_ips`/`matrix_coturn_denied_peer_ips` is also # possible for achieving such isolation, but is more complicated due to the dynamic nature of Docker networking. +# +# Setting `matrix_coturn_docker_network` to 'host' will run the container with host networking, +# which will drastically improve performance when thousands of ports are opened due to Docker not having to set up forwarding rules for each port. +# Running with host networking can be dangerous, as it potentially exposes your local network and its services to Coturn peers. +# Regardless of the networking mode, we apply a deny list which via `matrix_coturn_denied_peer_ips`, +# which hopefully prevents access to such private network ranges. +# When running in host-networking mode, you need to adjust the firewall yourself, so that ports are opened. matrix_coturn_docker_network: "matrix-coturn" matrix_coturn_base_path: "{{ matrix_base_data_path }}/coturn" @@ -41,20 +48,20 @@ matrix_coturn_container_extra_arguments: [] # Controls whether the Coturn container exposes its plain STUN port (tcp/3478 and udp/3478 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:3478"), or empty string to not expose. -matrix_coturn_container_stun_plain_host_bind_port: '3478' +matrix_coturn_container_stun_plain_host_bind_port: "{{ '3478' if matrix_coturn_docker_network != 'host' else '' }}" # Controls whether the Coturn container exposes its TLS STUN port (tcp/5349 and udp/5349 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:5349"), or empty string to not expose. -matrix_coturn_container_stun_tls_host_bind_port: '5349' +matrix_coturn_container_stun_tls_host_bind_port: "{{ '5349' if matrix_coturn_docker_network != 'host' else '' }}" # Controls whether the Coturn container exposes its TURN UDP port range and which interface to do it on. # # Takes an interface "" (e.g. "127.0.0.1"), or empty string to listen on all interfaces. -# Takes a null/none value (`~`) to prevent listening. +# Takes a null/none value (`~`) or 'none' (as a string) to prevent listening. # # The UDP port-range itself is specified using `matrix_coturn_turn_udp_min_port` and `matrix_coturn_turn_udp_max_port`. -matrix_coturn_container_turn_range_listen_interface: '' +matrix_coturn_container_turn_range_listen_interface: "{{ '' if matrix_coturn_docker_network != 'host' else 'none' }}" # UDP port-range to use for TURN matrix_coturn_turn_udp_min_port: 49152 diff --git a/roles/custom/matrix-coturn/tasks/setup_install.yml b/roles/custom/matrix-coturn/tasks/setup_install.yml index ab94db27..44325f46 100644 --- a/roles/custom/matrix-coturn/tasks/setup_install.yml +++ b/roles/custom/matrix-coturn/tasks/setup_install.yml @@ -62,7 +62,8 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" -- name: Ensure Coturn network is created in Docker +- when: matrix_coturn_docker_network not in ['', 'host'] + name: Ensure Coturn network is created in Docker community.docker.docker_network: name: "{{ matrix_coturn_docker_network }}" driver: bridge diff --git a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 index 3a0782bf..8c0272cf 100644 --- a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 +++ b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 @@ -30,7 +30,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name -p {{ matrix_coturn_container_stun_tls_host_bind_port }}:5349 \ -p {{ matrix_coturn_container_stun_tls_host_bind_port }}:5349/udp \ {% endif %} - {% if matrix_coturn_container_turn_range_listen_interface is not none %} + {% if matrix_coturn_container_turn_range_listen_interface is not in [none, 'none'] %} -p {{ matrix_coturn_container_turn_range_listen_interface }}{{ ':' if matrix_coturn_container_turn_range_listen_interface else '' }}{{ matrix_coturn_turn_udp_min_port }}-{{ matrix_coturn_turn_udp_max_port }}:{{ matrix_coturn_turn_udp_min_port }}-{{ matrix_coturn_turn_udp_max_port }}/udp \ {% endif %} --mount type=bind,src={{ matrix_coturn_config_path }},dst=/turnserver.conf,ro \ From 867737fe0b5bd9201c0d85ad3bcf3f4614823a50 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 17:42:03 +0200 Subject: [PATCH 015/382] Upgrade Grafana (9.3.4 -> 9.3.6) --- roles/custom/matrix-grafana/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-grafana/defaults/main.yml b/roles/custom/matrix-grafana/defaults/main.yml index 9920572d..5b730480 100644 --- a/roles/custom/matrix-grafana/defaults/main.yml +++ b/roles/custom/matrix-grafana/defaults/main.yml @@ -5,7 +5,7 @@ matrix_grafana_enabled: true -matrix_grafana_version: 9.3.4 +matrix_grafana_version: 9.3.6 matrix_grafana_docker_image: "{{ matrix_container_global_registry_prefix }}grafana/grafana:{{ matrix_grafana_version }}" matrix_grafana_docker_image_force_pull: "{{ matrix_grafana_docker_image.endswith(':latest') }}" From f1c9052fe530c41dcd24284765329f32029df221 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 17:59:56 +0200 Subject: [PATCH 016/382] Make it obvious that running Coturn in host-networking mode requires manual firewall changes --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 030fbbdb..23275a14 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,8 +10,8 @@ Such deployments don't need to run Coturn within a private container network any matrix_coturn_docker_network: host ``` -With such a configuration, Docker no longer needs to configure thousands of firewall forwarding rules each time Coturn starts and stops. -You may, however, need to allow these ports in your firewall configuration yourself. +With such a configuration, **Docker no longer needs to configure thousands of firewall forwarding rules** each time Coturn starts and stops. +This, however, means that **you will need to ensure these ports are open** in your firewall yourself. Thanks to us [tightening Coturn security](#backward-compatibility-tightening-coturn-security-can-lead-to-connectivity-issues), running Coturn with host-networking should be safe and not expose neither other services running on the host, nor other services running on the local network. From 864b64d0157f01e5240f1e6ea9de4c4215139c1d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 26 Jan 2023 20:01:38 +0200 Subject: [PATCH 017/382] Mention the new no-multicast-peers default in Coturn config Related to 4c9f96722f3f6ed56 --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 23275a14..edb4dba9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ Inspired by [this security article](https://www.rtcsec.com/article/cve-2020-2626 If you access your Matrix server from a local network and need Coturn to relay to private IP addresses, you may observe that relaying is now blocked due to our new default `denied-peer-ip` lists (specified in `matrix_coturn_denied_peer_ips`). If you experience such connectivity problems, consider overriding this setting in your `vars.yml` file and removing certain networks from it. +We've also added `no-multicast-peers` to the default Coturn configuration, but we don't expect this to cause trouble for most people. + # 2023-01-21 From e43ba65dd334fb9aadc74b94e4337080751a7fe7 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 27 Jan 2023 08:46:16 +0200 Subject: [PATCH 018/382] Upgrade prometheus_node_exporter (v1.5.0-0 -> v1.5.0-1) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 07cab197..8ffa99d5 100644 --- a/requirements.yml +++ b/requirements.yml @@ -31,4 +31,4 @@ version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git - version: v1.5.0-0 + version: v1.5.0-1 From 784043cc5dd0f6fef50004d40236ece31c7a4fa2 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 27 Jan 2023 08:55:38 +0200 Subject: [PATCH 019/382] Ensure OPENAI_PRO is true/false (not True/False) --- roles/custom/matrix-bot-chatgpt/templates/env.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-chatgpt/templates/env.j2 b/roles/custom/matrix-bot-chatgpt/templates/env.j2 index bf2dafbf..1315ddf5 100644 --- a/roles/custom/matrix-bot-chatgpt/templates/env.j2 +++ b/roles/custom/matrix-bot-chatgpt/templates/env.j2 @@ -4,7 +4,7 @@ MATRIX_ACCESS_TOKEN={{ matrix_bot_chatgpt_matrix_access_token }} OPENAI_EMAIL={{ matrix_bot_chatgpt_openai_email }} OPENAI_PASSWORD={{ matrix_bot_chatgpt_openai_password }} OPENAI_LOGIN_TYPE={{ matrix_bot_chatgpt_openai_login_type }} -OPENAI_PRO={{ matrix_bot_chatgpt_openai_pro }} +OPENAI_PRO={{ matrix_bot_chatgpt_openai_pro | to_json }} # With the @ and :DOMAIN, ie @SOMETHING:DOMAIN MATRIX_BOT_USERNAME={{ matrix_bot_chatgpt_matrix_bot_username }} From 7e1408ea65c2aaf162c887087c9ed403d615fcde Mon Sep 17 00:00:00 2001 From: bertybuttface <110790513+bertybuttface@users.noreply.github.com> Date: Fri, 27 Jan 2023 18:55:13 +0000 Subject: [PATCH 020/382] Bump ChatGPT version and add new config settings matrix_bot_chatgpt_context=thread --- roles/custom/matrix-bot-chatgpt/defaults/main.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index 86d185b7..3ba8ba7a 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_chatgpt_docker_repo: "https://github.com/matrixgpt/matrix-chatgpt-bot matrix_bot_chatgpt_docker_repo_version: "{{ 'latest' if matrix_bot_chatgpt_version == 'latest' else matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_src_files_path: "{{ matrix_base_data_path }}/chatgpt/docker-src" -matrix_bot_chatgpt_version: 1.3.2 +matrix_bot_chatgpt_version: 1.4.1 matrix_bot_chatgpt_docker_image: "{{ matrix_bot_chatgpt_docker_image_name_prefix }}matrixgpt/matrix-chatgpt-bot:{{ matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_chatgpt_container_image_self_build else 'ghcr.io/' }}" matrix_bot_chatgpt_docker_image_force_pull: "{{ matrix_bot_chatgpt_docker_image.endswith(':latest') }}" @@ -33,6 +33,8 @@ matrix_bot_chatgpt_openai_password: '' # OPENAI_PASSWORD= matrix_bot_chatgpt_openai_login_type: 'google' # OPENAI_LOGIN_TYPE="google" matrix_bot_chatgpt_openai_pro: false # OPENAI_PRO +matrix_bot_chatgpt_context: 'thread' # CHATGPT_CONTEXT="thread" + # Matrix Static Settings (required, see notes) # Defaults to "https://matrix.org" matrix_bot_chatgpt_matrix_homeserver_url: "{{ matrix_homeserver_container_url }}" # MATRIX_HOMESERVER_URL= From d66a2949f6b6a3e09a50f4da45f0d832b898d205 Mon Sep 17 00:00:00 2001 From: bertybuttface <110790513+bertybuttface@users.noreply.github.com> Date: Fri, 27 Jan 2023 18:57:24 +0000 Subject: [PATCH 021/382] Update env.j2 Use matrix_bot_chatgpt_context for CHATGPT_CONTEXT --- roles/custom/matrix-bot-chatgpt/templates/env.j2 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/roles/custom/matrix-bot-chatgpt/templates/env.j2 b/roles/custom/matrix-bot-chatgpt/templates/env.j2 index 1315ddf5..faf6be71 100644 --- a/roles/custom/matrix-bot-chatgpt/templates/env.j2 +++ b/roles/custom/matrix-bot-chatgpt/templates/env.j2 @@ -6,6 +6,8 @@ OPENAI_PASSWORD={{ matrix_bot_chatgpt_openai_password }} OPENAI_LOGIN_TYPE={{ matrix_bot_chatgpt_openai_login_type }} OPENAI_PRO={{ matrix_bot_chatgpt_openai_pro | to_json }} +CHATGPT_CONTEXT={{ matrix_bot_chatgpt_context }} + # With the @ and :DOMAIN, ie @SOMETHING:DOMAIN MATRIX_BOT_USERNAME={{ matrix_bot_chatgpt_matrix_bot_username }} MATRIX_BOT_PASSWORD={{ matrix_bot_chatgpt_matrix_bot_password }} From 7b5ced3037e16613f040d1044cf93a964eb3c439 Mon Sep 17 00:00:00 2001 From: bertybuttface <110790513+bertybuttface@users.noreply.github.com> Date: Fri, 27 Jan 2023 19:02:14 +0000 Subject: [PATCH 022/382] Fix linting issue. --- roles/custom/matrix-bot-chatgpt/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index 3ba8ba7a..324e64ab 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -33,7 +33,7 @@ matrix_bot_chatgpt_openai_password: '' # OPENAI_PASSWORD= matrix_bot_chatgpt_openai_login_type: 'google' # OPENAI_LOGIN_TYPE="google" matrix_bot_chatgpt_openai_pro: false # OPENAI_PRO -matrix_bot_chatgpt_context: 'thread' # CHATGPT_CONTEXT="thread" +matrix_bot_chatgpt_context: 'thread' # CHATGPT_CONTEXT="thread" # Matrix Static Settings (required, see notes) # Defaults to "https://matrix.org" From f165aa5d481e048089cb0b367ef61036dd893643 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sat, 28 Jan 2023 07:54:52 +0000 Subject: [PATCH 023/382] Update synapse-admin 0.8.5 -> 0.8.6 --- roles/custom/matrix-synapse-admin/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 9f4510b6..ae77a570 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -9,7 +9,7 @@ matrix_synapse_admin_container_image_self_build_repo: "https://github.com/Awesom matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src" -matrix_synapse_admin_version: 0.8.5 +matrix_synapse_admin_version: 0.8.6 matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:{{ matrix_synapse_admin_version }}" matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}" From 6a31fba346d9da434a3d7cfd2d342658d002cc2e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 28 Jan 2023 17:04:55 +0200 Subject: [PATCH 024/382] Fix synapse-admin reverse-proxying regression caused by 0.8.6 Related to f165aa5d481e048089cb0b367ef61036dd893643 Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2424 --- .../tasks/inject_into_nginx_proxy.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml index 6a4af859..b082190d 100644 --- a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml @@ -24,6 +24,14 @@ {# Generic configuration for use outside of our container setup #} proxy_pass http://127.0.0.1:8766/$1; {% endif %} + + {# + Workaround synapse-admin serving all assets at /static. + See: https://github.com/Awesome-Technologies/synapse-admin/issues/322 + #} + sub_filter_once off; + sub_filter_types text/css; + sub_filter "/static/" "{{ matrix_synapse_admin_public_endpoint }}/static/"; } - name: Register Synapse Admin proxying configuration with matrix-nginx-proxy From e588c42088c6fc5f94bb14d2473074294decc3b2 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 28 Jan 2023 17:09:20 +0200 Subject: [PATCH 025/382] Improve synapse-admin reverse-proxying fix Fixup for 6a31fba346d9da434a3d7cf --- .../matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml index b082190d..2c25ff47 100644 --- a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml @@ -30,8 +30,9 @@ See: https://github.com/Awesome-Technologies/synapse-admin/issues/322 #} sub_filter_once off; - sub_filter_types text/css; sub_filter "/static/" "{{ matrix_synapse_admin_public_endpoint }}/static/"; + sub_filter "/favicon.ico" "{{ matrix_synapse_admin_public_endpoint }}/favicon.ico"; + sub_filter "/manifest.json" "{{ matrix_synapse_admin_public_endpoint }}/manifest.json"; } - name: Register Synapse Admin proxying configuration with matrix-nginx-proxy From 691ef13cabd18c3f45f57ec6741a994fa62457ee Mon Sep 17 00:00:00 2001 From: Adrien le Maire <1832823+alemairebe@users.noreply.github.com> Date: Sun, 29 Jan 2023 09:45:52 +0100 Subject: [PATCH 026/382] template metric toggle for mautrix bridges supporting it --- roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml | 3 +++ .../matrix-bridge-mautrix-facebook/templates/config.yaml.j2 | 2 +- .../custom/matrix-bridge-mautrix-instagram/defaults/main.yml | 3 +++ .../matrix-bridge-mautrix-instagram/templates/config.yaml.j2 | 2 +- roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml | 3 +++ .../matrix-bridge-mautrix-signal/templates/config.yaml.j2 | 2 +- roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml | 3 +++ .../matrix-bridge-mautrix-twitter/templates/config.yaml.j2 | 2 +- roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml | 4 ++++ .../matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 | 4 ++-- 10 files changed, 22 insertions(+), 6 deletions(-) diff --git a/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml index 98972014..7863270b 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml @@ -46,6 +46,9 @@ matrix_mautrix_facebook_homeserver_token: '' # If false, created portal rooms will never be federated. matrix_mautrix_facebook_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +matrix_mautrix_facebook_metrics_enabled: false + matrix_mautrix_facebook_bridge_permissions: | {{ {matrix_mautrix_facebook_homeserver_domain: 'user'} diff --git a/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 index b8e25fee..ce9aaea3 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 @@ -62,7 +62,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_facebook_metrics_enabled }} listen_port: 8000 # Bridge config diff --git a/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml index 0e726aa8..e9b6991b 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml @@ -47,6 +47,9 @@ matrix_mautrix_instagram_homeserver_token: '' # If false, created portal rooms will never be federated. matrix_mautrix_instagram_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +matrix_mautrix_instagram_metrics_enabled: false + # Database-related configuration fields. # # To use Postgres: diff --git a/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 index 039b9bfe..70065785 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 @@ -57,7 +57,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_instagram_metrics_enabled }} listen_port: 8000 instagram: diff --git a/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml index 953d1707..fb5e49b1 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml @@ -70,6 +70,9 @@ matrix_mautrix_signal_logging_level: WARNING # If false, created portal rooms will never be federated. matrix_mautrix_signal_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +matrix_mautrix_signal_metrics_enabled: false + # Database-related configuration fields # # This bridge only supports postgres. diff --git a/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 index 796a6e41..fbe2f4f3 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 @@ -67,7 +67,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_signal_metrics_enabled }} listen_port: 8000 # Manhole config. diff --git a/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml index 22475596..372dd4b6 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml @@ -78,6 +78,9 @@ matrix_mautrix_twitter_appservice_bot_username: twitterbot # Specifies the default log level for all bridge loggers. matrix_mautrix_twitter_logging_level: WARNING +# Whether or not metrics endpoint should be enabled. +matrix_mautrix_twitter_metrics_enabled: false + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 index da823d1e..5b79a073 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 @@ -65,7 +65,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_twitter_metrics_enabled }} listen_port: 8000 # Bridge config diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml index 449ace05..91e27dc0 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -47,6 +47,10 @@ matrix_mautrix_whatsapp_logging_level: 'warn' # If false, created portal rooms will never be federated. matrix_mautrix_whatsapp_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +matrix_mautrix_whatsapp_metrics_enabled: false + + # Database-related configuration fields. # # To use SQLite, stick to these defaults. diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 index 0962ac4d..1c6b59d4 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 @@ -67,9 +67,9 @@ segment_key: null # Prometheus config. metrics: # Enable prometheus metrics? - enabled: false + enabled: {{ matrix_mautrix_whatsapp_metrics_enabled }} # IP and port where the metrics listener should be. The path is always /metrics - listen: 127.0.0.1:8001 + listen: 0.0.0.0:8001 # Config for things that are directly sent to WhatsApp. whatsapp: From 9eaf6944e3c6d8ea8f1094dc515713053e190573 Mon Sep 17 00:00:00 2001 From: Adrien le Maire <1832823+alemairebe@users.noreply.github.com> Date: Sun, 29 Jan 2023 15:13:42 +0100 Subject: [PATCH 027/382] add nginx proxy connfig for mautrix metrics --- .../tasks/inject_into_nginx_proxy.yml | 25 ++++++++++++++ .../tasks/inject_into_nginx_proxy.yml | 33 +++++++++++++++++++ .../tasks/main.yml | 9 +++++ .../tasks/inject_into_nginx_proxy.yml | 33 +++++++++++++++++++ .../tasks/main.yml | 9 +++++ .../tasks/inject_into_nginx_proxy.yml | 33 +++++++++++++++++++ .../tasks/main.yml | 9 +++++ .../defaults/main.yml | 1 - .../tasks/inject_into_nginx_proxy.yml | 33 +++++++++++++++++++ .../tasks/main.yml | 9 +++++ 10 files changed, 193 insertions(+), 1 deletion(-) create mode 100644 roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml create mode 100644 roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml create mode 100644 roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml create mode 100644 roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml diff --git a/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml index b9f95cb7..db4d906f 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml @@ -42,3 +42,28 @@ URL endpoint to the matrix-mautrix-facebook container. You can expose the container's port using the `matrix_mautrix_facebook_container_http_host_bind_port` variable. when: "not matrix_nginx_proxy_enabled | default(False) | bool" + +- name: Generate mautrix-facebook metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) + ansible.builtin.set_fact: + matrix_mautrix_facebook_nginx_metrics_configuration_block: | + location /metrics/mautrix-facebook { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-facebook:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + when: matrix_mautrix_facebook_metrics_enabled | bool + +- name: Register mautrix-facebook metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_facebook_nginx_metrics_configuration_block] + }} + when: matrix_mautrix_facebook_metrics_enabled | bool diff --git a/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..90d91c6d --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,33 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-instagram role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- name: Generate mautrix-instagram metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) + ansible.builtin.set_fact: + matrix_mautrix_instagram_nginx_metrics_configuration_block: | + location /metrics/mautrix-instagram { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-instagram:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + +- name: Register mautrix-instagram metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_instagram_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml index 2bd0417e..0b3a722d 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_instagram_enabled | bool and matrix_mautrix_instagram_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" diff --git a/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..4bb6f6bd --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,33 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-signal role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- name: Generate mautrix-signal metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) + ansible.builtin.set_fact: + matrix_mautrix_signal_nginx_metrics_configuration_block: | + location /metrics/mautrix-signal { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-signal:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + +- name: Register mautrix-signal metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_signal_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml index 0c0a1239..9473344b 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_signal_enabled | bool and matrix_mautrix_signal_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_signal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" diff --git a/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..df296dc6 --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,33 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-twitter role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- name: Generate mautrix-twitter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) + ansible.builtin.set_fact: + matrix_mautrix_twitter_nginx_metrics_configuration_block: | + location /metrics/mautrix-twitter { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-twitter:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + +- name: Register mautrix-twitter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_twitter_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml index f8dbc28b..9eb444a7 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_twitter_enabled | bool and matrix_mautrix_twitter_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml index 91e27dc0..6e85599a 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -50,7 +50,6 @@ matrix_mautrix_whatsapp_federate_rooms: true # Whether or not metrics endpoint should be enabled. matrix_mautrix_whatsapp_metrics_enabled: false - # Database-related configuration fields. # # To use SQLite, stick to these defaults. diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..e3fc3d16 --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,33 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-whatsapp role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- name: Generate mautrix-whatsapp metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) + ansible.builtin.set_fact: + matrix_mautrix_whatsapp_nginx_metrics_configuration_block: | + location /metrics/mautrix-whatsapp { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-whatsapp:8001"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + +- name: Register mautrix-whatsapp metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_whatsapp_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml index fdb65431..b49a1300 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_whatsapp_enabled | bool and matrix_mautrix_whatsapp_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_whatsapp_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" From 111303208a94131e285bbeda43befde2bdfc41df Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 29 Jan 2023 18:11:33 +0000 Subject: [PATCH 028/382] Update mautrix-discord latest -> 0.1.0 --- roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml index a66d5c6a..0211f8f3 100644 --- a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_discord_container_image_self_build: false matrix_mautrix_discord_container_image_self_build_repo: "https://mau.dev/mautrix/discord.git" matrix_mautrix_discord_container_image_self_build_branch: "{{ 'main' if matrix_mautrix_discord_version == 'latest' else matrix_mautrix_discord_version }}" -matrix_mautrix_discord_version: latest +matrix_mautrix_discord_version: v0.1.0 # See: https://mau.dev/mautrix/discord/container_registry matrix_mautrix_discord_docker_image: "{{ matrix_mautrix_discord_docker_image_name_prefix }}mautrix/discord:{{ matrix_mautrix_discord_version }}" matrix_mautrix_discord_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_discord_container_image_self_build else 'dock.mau.dev/' }}" From a9a17d803e2436b64b95ba5c70029b37a9e957f1 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 29 Jan 2023 18:12:26 +0000 Subject: [PATCH 029/382] Update maubot 0.3.1 -> 0.4.0 --- roles/custom/matrix-bot-maubot/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-maubot/defaults/main.yml b/roles/custom/matrix-bot-maubot/defaults/main.yml index 525b5e9e..caee9b13 100644 --- a/roles/custom/matrix-bot-maubot/defaults/main.yml +++ b/roles/custom/matrix-bot-maubot/defaults/main.yml @@ -10,7 +10,7 @@ matrix_bot_maubot_docker_src_files_path: "{{ matrix_bot_maubot_base_path }}/dock matrix_bot_maubot_docker_repo_version: "{{ 'master' if matrix_bot_maubot_version == 'latest' else matrix_bot_maubot_version }}" -matrix_bot_maubot_version: v0.3.1 +matrix_bot_maubot_version: v0.4.0 matrix_bot_maubot_docker_image: "{{ matrix_bot_maubot_docker_image_name_prefix }}maubot/maubot:{{ matrix_bot_maubot_version }}" matrix_bot_maubot_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_maubot_container_image_self_build else 'dock.mau.dev/' }}" matrix_bot_maubot_docker_image_force_pull: "{{ matrix_bot_maubot_docker_image.endswith(':latest') }}" From 4f69b22a6ec72b06e004f33936c21945dad3b2ae Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 29 Jan 2023 18:14:41 +0000 Subject: [PATCH 030/382] Update borgmatic 1.7.5 -> 1.7.6 --- roles/custom/matrix-backup-borg/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-backup-borg/defaults/main.yml b/roles/custom/matrix-backup-borg/defaults/main.yml index b8796c92..ad5a0281 100644 --- a/roles/custom/matrix-backup-borg/defaults/main.yml +++ b/roles/custom/matrix-backup-borg/defaults/main.yml @@ -15,7 +15,7 @@ matrix_backup_borg_docker_src_files_path: "{{ matrix_backup_borg_base_path }}/do matrix_backup_borg_version: "" matrix_backup_borg_postgres_version: "" matrix_backup_borg_borg_version: 1.2.3 -matrix_backup_borg_borgmatic_version: 1.7.5 +matrix_backup_borg_borgmatic_version: 1.7.6 matrix_backup_borg_docker_image: "{{ matrix_backup_borg_docker_image_name_prefix }}etke.cc/borgmatic:{{ matrix_backup_borg_version }}" matrix_backup_borg_docker_image_name_prefix: "{{ 'localhost/' if matrix_backup_borg_container_image_self_build else 'registry.gitlab.com/' }}" matrix_backup_borg_docker_image_force_pull: "{{ matrix_backup_borg_docker_image.endswith(':latest') or matrix_backup_borg_version | default('') == '' }}" From df9931f719e3e8252a2c399ba9062d0ec952371f Mon Sep 17 00:00:00 2001 From: Shaleen Jain Date: Mon, 30 Jan 2023 04:45:31 +0000 Subject: [PATCH 031/382] mautrix-slack: add team name in channel name template --- .../custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 index 1c7a1dff..5c60c66f 100644 --- a/roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 @@ -79,7 +79,7 @@ bridge: # TODO: document variables displayname_template: "{{ '{{.RealName}} (S)' }}" bot_displayname_template: "{{ '{{.Name}} (bot)' }}" - channel_name_template: "{{ '#{{.Name}}' }}" + channel_name_template: "{{ '#{{.Name}} ({{.TeamName}})' }}" portal_message_buffer: 128 From d82d0ad84bb4cf63bb0bb04fa769d57d0f831eb5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 30 Jan 2023 08:50:57 +0200 Subject: [PATCH 032/382] Add _metrics_proxying_enabled variables to mautrix bridges Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2427 `metrics_enabled` should only expose the metrics locally, on the container network, so that a local Prometheus can consume them. Exposing them publicly should be done via a separate toggle (`metrics_proxying_enabled`). This is how all other roles work, so this makes these mautrix roles consistent with the rest. --- .../defaults/main.yml | 7 +++ .../tasks/inject_into_nginx_proxy.yml | 46 +++++++++---------- .../defaults/main.yml | 7 +++ .../tasks/inject_into_nginx_proxy.yml | 44 +++++++++--------- .../defaults/main.yml | 7 +++ .../tasks/inject_into_nginx_proxy.yml | 44 +++++++++--------- .../defaults/main.yml | 7 +++ .../tasks/inject_into_nginx_proxy.yml | 44 +++++++++--------- .../defaults/main.yml | 7 +++ .../tasks/inject_into_nginx_proxy.yml | 44 +++++++++--------- 10 files changed, 150 insertions(+), 107 deletions(-) diff --git a/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml index 7863270b..8c19139a 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml @@ -47,8 +47,15 @@ matrix_mautrix_facebook_homeserver_token: '' matrix_mautrix_facebook_federate_rooms: true # Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_facebook_metrics_proxying_enabled`. matrix_mautrix_facebook_metrics_enabled: false +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-facebook`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_facebook_metrics_proxying_enabled: false + matrix_mautrix_facebook_bridge_permissions: | {{ {matrix_mautrix_facebook_homeserver_domain: 'user'} diff --git a/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml index db4d906f..17e2b138 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml @@ -43,27 +43,27 @@ You can expose the container's port using the `matrix_mautrix_facebook_container_http_host_bind_port` variable. when: "not matrix_nginx_proxy_enabled | default(False) | bool" -- name: Generate mautrix-facebook metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) - ansible.builtin.set_fact: - matrix_mautrix_facebook_nginx_metrics_configuration_block: | - location /metrics/mautrix-facebook { - {% if matrix_nginx_proxy_enabled | default(False) %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "matrix-mautrix-facebook:8000"; - proxy_pass http://$backend/metrics; - {% else %} - return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; - {% endif %} - } - when: matrix_mautrix_facebook_metrics_enabled | bool +- when: matrix_mautrix_facebook_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-facebook metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) + ansible.builtin.set_fact: + matrix_mautrix_facebook_nginx_metrics_configuration_block: | + location /metrics/mautrix-facebook { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-facebook:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } -- name: Register mautrix-facebook metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) - ansible.builtin.set_fact: - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | - {{ - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) - + - [matrix_mautrix_facebook_nginx_metrics_configuration_block] - }} - when: matrix_mautrix_facebook_metrics_enabled | bool + - name: Register mautrix-facebook metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_facebook_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml index e9b6991b..80d59162 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml @@ -48,8 +48,15 @@ matrix_mautrix_instagram_homeserver_token: '' matrix_mautrix_instagram_federate_rooms: true # Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_instagram_metrics_proxying_enabled`. matrix_mautrix_instagram_metrics_enabled: false +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-instagram`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_instagram_metrics_proxying_enabled: false + # Database-related configuration fields. # # To use Postgres: diff --git a/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml index 90d91c6d..99a36c14 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml @@ -9,25 +9,27 @@ so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-instagram role. when: matrix_nginx_proxy_role_executed | default(False) | bool -- name: Generate mautrix-instagram metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) - ansible.builtin.set_fact: - matrix_mautrix_instagram_nginx_metrics_configuration_block: | - location /metrics/mautrix-instagram { - {% if matrix_nginx_proxy_enabled | default(False) %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "matrix-mautrix-instagram:8000"; - proxy_pass http://$backend/metrics; - {% else %} - return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; - {% endif %} - } +- when: matrix_mautrix_instagram_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-instagram metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) + ansible.builtin.set_fact: + matrix_mautrix_instagram_nginx_metrics_configuration_block: | + location /metrics/mautrix-instagram { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-instagram:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } -- name: Register mautrix-instagram metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) - ansible.builtin.set_fact: - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | - {{ - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) - + - [matrix_mautrix_instagram_nginx_metrics_configuration_block] - }} + - name: Register mautrix-instagram metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_instagram_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml index fb5e49b1..8cd285ca 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml @@ -71,8 +71,15 @@ matrix_mautrix_signal_logging_level: WARNING matrix_mautrix_signal_federate_rooms: true # Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_signal_metrics_proxying_enabled`. matrix_mautrix_signal_metrics_enabled: false +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-signal`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_signal_metrics_proxying_enabled: false + # Database-related configuration fields # # This bridge only supports postgres. diff --git a/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml index 4bb6f6bd..fe85254b 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml @@ -9,25 +9,27 @@ so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-signal role. when: matrix_nginx_proxy_role_executed | default(False) | bool -- name: Generate mautrix-signal metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) - ansible.builtin.set_fact: - matrix_mautrix_signal_nginx_metrics_configuration_block: | - location /metrics/mautrix-signal { - {% if matrix_nginx_proxy_enabled | default(False) %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "matrix-mautrix-signal:8000"; - proxy_pass http://$backend/metrics; - {% else %} - return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; - {% endif %} - } +- when: matrix_mautrix_signal_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-signal metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) + ansible.builtin.set_fact: + matrix_mautrix_signal_nginx_metrics_configuration_block: | + location /metrics/mautrix-signal { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-signal:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } -- name: Register mautrix-signal metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) - ansible.builtin.set_fact: - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | - {{ - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) - + - [matrix_mautrix_signal_nginx_metrics_configuration_block] - }} + - name: Register mautrix-signal metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_signal_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml index 372dd4b6..8f38bb00 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml @@ -79,8 +79,15 @@ matrix_mautrix_twitter_appservice_bot_username: twitterbot matrix_mautrix_twitter_logging_level: WARNING # Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_twitter_metrics_proxying_enabled`. matrix_mautrix_twitter_metrics_enabled: false +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-twitter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_twitter_metrics_proxying_enabled: false + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml index df296dc6..2efad7cd 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml @@ -9,25 +9,27 @@ so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-twitter role. when: matrix_nginx_proxy_role_executed | default(False) | bool -- name: Generate mautrix-twitter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) - ansible.builtin.set_fact: - matrix_mautrix_twitter_nginx_metrics_configuration_block: | - location /metrics/mautrix-twitter { - {% if matrix_nginx_proxy_enabled | default(False) %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "matrix-mautrix-twitter:8000"; - proxy_pass http://$backend/metrics; - {% else %} - return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; - {% endif %} - } +- when: matrix_mautrix_twitter_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-twitter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) + ansible.builtin.set_fact: + matrix_mautrix_twitter_nginx_metrics_configuration_block: | + location /metrics/mautrix-twitter { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-twitter:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } -- name: Register mautrix-twitter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) - ansible.builtin.set_fact: - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | - {{ - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) - + - [matrix_mautrix_twitter_nginx_metrics_configuration_block] - }} + - name: Register mautrix-twitter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_twitter_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml index 6e85599a..dd628162 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -48,8 +48,15 @@ matrix_mautrix_whatsapp_logging_level: 'warn' matrix_mautrix_whatsapp_federate_rooms: true # Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_whatsapp_metrics_proxying_enabled`. matrix_mautrix_whatsapp_metrics_enabled: false +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-whatsapp`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_whatsapp_metrics_proxying_enabled: false + # Database-related configuration fields. # # To use SQLite, stick to these defaults. diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml index e3fc3d16..03113dcc 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml @@ -9,25 +9,27 @@ so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-whatsapp role. when: matrix_nginx_proxy_role_executed | default(False) | bool -- name: Generate mautrix-whatsapp metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) - ansible.builtin.set_fact: - matrix_mautrix_whatsapp_nginx_metrics_configuration_block: | - location /metrics/mautrix-whatsapp { - {% if matrix_nginx_proxy_enabled | default(False) %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "matrix-mautrix-whatsapp:8001"; - proxy_pass http://$backend/metrics; - {% else %} - return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; - {% endif %} - } +- when: matrix_mautrix_whatsapp_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-whatsapp metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) + ansible.builtin.set_fact: + matrix_mautrix_whatsapp_nginx_metrics_configuration_block: | + location /metrics/mautrix-whatsapp { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-whatsapp:8001"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } -- name: Register mautrix-whatsapp metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) - ansible.builtin.set_fact: - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | - {{ - matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) - + - [matrix_mautrix_whatsapp_nginx_metrics_configuration_block] - }} + - name: Register mautrix-whatsapp metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_whatsapp_nginx_metrics_configuration_block] + }} From d7c0239e40f8a6b0bec1266055cf02dcca29d98f Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 30 Jan 2023 08:53:28 +0200 Subject: [PATCH 033/382] Enable metrics endpoint for mautrix bridges by default when Prometheus is enabled Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2427 This just enables the endpoint, which is somewhat helpful, but not really enough to scrape them. Ideally, we'd be injecting these targets into the Prometheus scrape config too. For now, registering targets with Prometheus is very manual (`matrix_prometheus_scraper_postgres_enabled`, `matrix_prometheus_scraper_hookshot_enabled`, ..). This should be redone - e.g. a new `matrix_prometheus_scrape_config_jobs_auto` variable, which is dynamically built in `group_vars/matrix_servers`. --- group_vars/matrix_servers | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 70abde3a..2a8b2443 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -807,6 +807,12 @@ matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_pro matrix_mautrix_facebook_bridge_presence: "{{ matrix_synapse_presence_enabled if matrix_synapse_enabled else true }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_facebook_metrics_enabled` +# - `matrix_mautrix_facebook_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_facebook_metrics_enabled: "{{ matrix_prometheus_enabled }}" + # We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain # and point them to a migration path. matrix_mautrix_facebook_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -934,6 +940,12 @@ matrix_mautrix_instagram_login_shared_secret: "{{ matrix_synapse_ext_password_pr matrix_mautrix_instagram_bridge_presence: "{{ matrix_synapse_presence_enabled if matrix_synapse_enabled else true }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_instagram_metrics_enabled` +# - `matrix_mautrix_instagram_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_instagram_metrics_enabled: "{{ matrix_prometheus_enabled }}" + # We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain # and point them to a migration path. matrix_mautrix_instagram_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -979,6 +991,12 @@ matrix_mautrix_signal_appservice_token: "{{ '%s' | format(matrix_homeserver_gene matrix_mautrix_signal_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_signal_metrics_enabled` +# - `matrix_mautrix_signal_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_signal_metrics_enabled: "{{ matrix_prometheus_enabled }}" + matrix_mautrix_signal_database_engine: 'postgres' matrix_mautrix_signal_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" matrix_mautrix_signal_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'mau.signal.db', rounds=655555) | to_uuid }}" @@ -1103,6 +1121,12 @@ matrix_mautrix_whatsapp_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_whatsapp_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_twitter_metrics_enabled` +# - `matrix_mautrix_twitter_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_twitter_metrics_enabled: "{{ matrix_prometheus_enabled }}" + # Postgres is the default, except if not using internal Postgres server matrix_mautrix_whatsapp_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" matrix_mautrix_whatsapp_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" From 611a74bde2414cb95171d4833c926a5427c08c21 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 30 Jan 2023 08:59:35 +0200 Subject: [PATCH 034/382] Use |to_json in mautrix metrics configuration Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2427 --- .../matrix-bridge-mautrix-facebook/templates/config.yaml.j2 | 2 +- .../matrix-bridge-mautrix-instagram/templates/config.yaml.j2 | 2 +- .../matrix-bridge-mautrix-signal/templates/config.yaml.j2 | 2 +- .../matrix-bridge-mautrix-twitter/templates/config.yaml.j2 | 2 +- .../matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 index ce9aaea3..184ea780 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 @@ -62,7 +62,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: {{ matrix_mautrix_facebook_metrics_enabled }} + enabled: {{ matrix_mautrix_facebook_metrics_enabled | to_json }} listen_port: 8000 # Bridge config diff --git a/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 index 70065785..93bbfe39 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 @@ -57,7 +57,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: {{ matrix_mautrix_instagram_metrics_enabled }} + enabled: {{ matrix_mautrix_instagram_metrics_enabled | to_json }} listen_port: 8000 instagram: diff --git a/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 index fbe2f4f3..828ff940 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 @@ -67,7 +67,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: {{ matrix_mautrix_signal_metrics_enabled }} + enabled: {{ matrix_mautrix_signal_metrics_enabled | to_json }} listen_port: 8000 # Manhole config. diff --git a/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 index 5b79a073..6e12a755 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 @@ -65,7 +65,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: {{ matrix_mautrix_twitter_metrics_enabled }} + enabled: {{ matrix_mautrix_twitter_metrics_enabled | to_json }} listen_port: 8000 # Bridge config diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 index 1c6b59d4..ad381f6a 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 @@ -67,7 +67,7 @@ segment_key: null # Prometheus config. metrics: # Enable prometheus metrics? - enabled: {{ matrix_mautrix_whatsapp_metrics_enabled }} + enabled: {{ matrix_mautrix_whatsapp_metrics_enabled | to_json }} # IP and port where the metrics listener should be. The path is always /metrics listen: 0.0.0.0:8001 From 8c2b77bf0cf5400f6234a3adcf97758f1bc62d93 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Mon, 30 Jan 2023 07:07:09 +0000 Subject: [PATCH 035/382] Update cinny 2.2.3 -> 2.2.4 --- roles/custom/matrix-client-cinny/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-cinny/defaults/main.yml b/roles/custom/matrix-client-cinny/defaults/main.yml index 98c32745..5b15b42c 100644 --- a/roles/custom/matrix-client-cinny/defaults/main.yml +++ b/roles/custom/matrix-client-cinny/defaults/main.yml @@ -6,7 +6,7 @@ matrix_client_cinny_enabled: true matrix_client_cinny_container_image_self_build: false matrix_client_cinny_container_image_self_build_repo: "https://github.com/ajbura/cinny.git" -matrix_client_cinny_version: v2.2.3 +matrix_client_cinny_version: v2.2.4 matrix_client_cinny_docker_image: "{{ matrix_client_cinny_docker_image_name_prefix }}ajbura/cinny:{{ matrix_client_cinny_version }}" matrix_client_cinny_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_cinny_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_cinny_docker_image_force_pull: "{{ matrix_client_cinny_docker_image.endswith(':latest') }}" From d70076c805bd30f7e69a35115d7e6ccdb772d88b Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 31 Jan 2023 14:01:11 +0000 Subject: [PATCH 036/382] Update element 1.11.20 -> 1.11.21 --- roles/custom/matrix-client-element/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index f13ad035..ea88e58f 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -10,7 +10,7 @@ matrix_client_element_container_image_self_build_repo: "https://github.com/vecto # - https://github.com/vector-im/element-web/issues/19544 matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}" -matrix_client_element_version: v1.11.20 +matrix_client_element_version: v1.11.21 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" From 68ca23d709c6025ade132466158715f87952debd Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 31 Jan 2023 14:02:50 +0000 Subject: [PATCH 037/382] Update jitsi stable-8218 -> stable-8252 --- roles/custom/matrix-jitsi/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index 538d8e4e..8ccd6793 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -72,7 +72,7 @@ matrix_jitsi_jibri_recorder_password: '' matrix_jitsi_enable_lobby: false -matrix_jitsi_version: stable-8218 +matrix_jitsi_version: stable-8252 matrix_jitsi_container_image_tag: "{{ matrix_jitsi_version }}" # for backward-compatibility matrix_jitsi_web_docker_image: "{{ matrix_container_global_registry_prefix }}jitsi/web:{{ matrix_jitsi_container_image_tag }}" From c7767e9bc80e0e94801560d2d661d466d0024175 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 31 Jan 2023 20:25:59 +0200 Subject: [PATCH 038/382] Upgrade Coturn (4.6.1-r0 -> 4.6.1-r1) --- roles/custom/matrix-coturn/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 9f7ff2ba..78efc73c 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -8,7 +8,7 @@ matrix_coturn_container_image_self_build_repo: "https://github.com/coturn/coturn matrix_coturn_container_image_self_build_repo_version: "docker/{{ matrix_coturn_version }}" matrix_coturn_container_image_self_build_repo_dockerfile_path: "docker/coturn/alpine/Dockerfile" -matrix_coturn_version: 4.6.1-r0 +matrix_coturn_version: 4.6.1-r1 matrix_coturn_docker_image: "{{ matrix_coturn_docker_image_name_prefix }}coturn/coturn:{{ matrix_coturn_version }}-alpine" matrix_coturn_docker_image_name_prefix: "{{ 'localhost/' if matrix_coturn_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_coturn_docker_image_force_pull: "{{ matrix_coturn_docker_image.endswith(':latest') }}" From 5300740f703e040bb3dab81fbcb98a5e3ab05a9e Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 31 Jan 2023 19:11:07 +0000 Subject: [PATCH 039/382] Update element 1.11.21 -> 1.11.22 --- roles/custom/matrix-client-element/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index ea88e58f..85d8d6cf 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -10,7 +10,7 @@ matrix_client_element_container_image_self_build_repo: "https://github.com/vecto # - https://github.com/vector-im/element-web/issues/19544 matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}" -matrix_client_element_version: v1.11.21 +matrix_client_element_version: v1.11.22 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" From 82d870fddf6c658df41128a705a5157e4a586aca Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 31 Jan 2023 19:16:52 +0000 Subject: [PATCH 040/382] Update prometheus 2.41.0 -> 2.42.0 --- roles/custom/matrix-prometheus/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-prometheus/defaults/main.yml b/roles/custom/matrix-prometheus/defaults/main.yml index 5216ccd7..956faf46 100644 --- a/roles/custom/matrix-prometheus/defaults/main.yml +++ b/roles/custom/matrix-prometheus/defaults/main.yml @@ -5,7 +5,7 @@ matrix_prometheus_enabled: false -matrix_prometheus_version: v2.41.0 +matrix_prometheus_version: v2.42.0 matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}" matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}" From 0f208ed053bc33dd451da9c4b6ed9a08b30a908b Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 31 Jan 2023 19:19:43 +0000 Subject: [PATCH 041/382] Update synapse 1.75.0 -> 1.76.0; default room version 9 -> 10 --- roles/custom/matrix-synapse/defaults/main.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 7c5496fa..b5a214ed 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -36,7 +36,7 @@ matrix_synapse_container_image_customizations_dockerfile_body_custom: '' matrix_synapse_docker_image: "{{ matrix_synapse_docker_image_name_prefix }}matrixdotorg/synapse:{{ matrix_synapse_docker_image_tag }}" matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_container_image_self_build else matrix_container_global_registry_prefix }}" -matrix_synapse_version: v1.75.0 +matrix_synapse_version: v1.76.0 matrix_synapse_docker_image_tag: "{{ matrix_synapse_version }}" matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}" @@ -849,7 +849,7 @@ matrix_synapse_room_list_publication_rules: room_id: "*" action: allow -matrix_synapse_default_room_version: "9" +matrix_synapse_default_room_version: "10" # Controls the Synapse `spam_checker` setting. # From 7cb140b98744890adbdb148b2b68f51594c33d98 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 31 Jan 2023 23:24:20 +0200 Subject: [PATCH 042/382] Downgrade Prometheus (v2.42.0 -> v2.41.0) until a container image gets published Container image not published yet. Reverts #2438 --- roles/custom/matrix-prometheus/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-prometheus/defaults/main.yml b/roles/custom/matrix-prometheus/defaults/main.yml index 956faf46..5216ccd7 100644 --- a/roles/custom/matrix-prometheus/defaults/main.yml +++ b/roles/custom/matrix-prometheus/defaults/main.yml @@ -5,7 +5,7 @@ matrix_prometheus_enabled: false -matrix_prometheus_version: v2.42.0 +matrix_prometheus_version: v2.41.0 matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}" matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}" From c11f772e781c762f69810e374ac3a4aaf5ff8362 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 31 Jan 2023 21:34:25 +0000 Subject: [PATCH 043/382] Fix python packages path in synapse container --- roles/custom/matrix-synapse/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index b5a214ed..18165dd9 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -123,7 +123,7 @@ matrix_synapse_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-synapse.service wants matrix_synapse_systemd_wanted_services_list: [] -matrix_synapse_in_container_python_packages_path: "/usr/local/lib/python3.9/site-packages" +matrix_synapse_in_container_python_packages_path: "/usr/local/lib/python3.11/site-packages" # Specifies which template files to use when configuring Synapse. # If you'd like to have your own different configuration, feel free to copy and paste From 4d49f1f56ecd5feddc83bc704ac950161dabe6dc Mon Sep 17 00:00:00 2001 From: Catalan Lover <48515417+FSG-Cat@users.noreply.github.com> Date: Thu, 2 Feb 2023 16:00:07 +0100 Subject: [PATCH 044/382] Update Prometheus to v2.42.0 from v2.41.0 Docker images are released now so this change can now be pushed. --- roles/custom/matrix-prometheus/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-prometheus/defaults/main.yml b/roles/custom/matrix-prometheus/defaults/main.yml index 5216ccd7..956faf46 100644 --- a/roles/custom/matrix-prometheus/defaults/main.yml +++ b/roles/custom/matrix-prometheus/defaults/main.yml @@ -5,7 +5,7 @@ matrix_prometheus_enabled: false -matrix_prometheus_version: v2.41.0 +matrix_prometheus_version: v2.42.0 matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}" matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}" From 2d7d5d4babd11d7ed70060b4d8d06b6113dc519a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 3 Feb 2023 20:36:19 +0200 Subject: [PATCH 045/382] Use new security-opt syntax (: -> =) Related to https://docs.docker.com/engine/deprecated/#separator--of---security-opt-flag-on-docker-run --- .../templates/goofys/systemd/matrix-goofys.service.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 b/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 index a1174bce..a968965b 100644 --- a/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 +++ b/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 @@ -17,7 +17,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name --mount type=bind,src=/etc/passwd,dst=/etc/passwd,ro \ --mount type=bind,src=/etc/group,dst=/etc/group,ro \ --mount type=bind,src={{ matrix_s3_media_store_path }},dst=/s3,bind-propagation=shared \ - --security-opt apparmor:unconfined \ + --security-opt apparmor=unconfined \ --cap-add mknod \ --cap-add sys_admin \ --device=/dev/fuse \ From be78b74fbdd00ab89b0827f7ddec91ac7e21aeb9 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 5 Feb 2023 10:32:09 +0200 Subject: [PATCH 046/382] Switch from matrix-prometheus-postgres-exporter to an external prometheus_postgres_exporter role --- CHANGELOG.md | 17 +++- ...configuring-playbook-prometheus-grafana.md | 8 +- ...onfiguring-playbook-prometheus-postgres.md | 10 +-- group_vars/matrix_servers | 89 ++++++++++--------- playbooks/matrix.yml | 2 +- requirements.yml | 3 + .../defaults/main.yml | 65 -------------- .../tasks/main.yml | 29 ------ .../tasks/setup_install.yml | 19 ---- .../tasks/setup_uninstall.yml | 20 ----- .../tasks/validate_config.yml | 9 -- ...ix-prometheus-postgres-exporter.service.j2 | 42 --------- .../vars/main.yml | 5 -- .../defaults/main.yml | 12 +++ .../tasks/main.yml | 9 ++ .../inject_into_nginx_proxy.yml | 19 ++-- .../defaults/main.yml | 3 + .../tasks/validate_config.yml | 16 ++++ 18 files changed, 123 insertions(+), 254 deletions(-) delete mode 100644 roles/custom/matrix-prometheus-postgres-exporter/defaults/main.yml delete mode 100644 roles/custom/matrix-prometheus-postgres-exporter/tasks/main.yml delete mode 100644 roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-prometheus-postgres-exporter/tasks/validate_config.yml delete mode 100644 roles/custom/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2 delete mode 100644 roles/custom/matrix-prometheus-postgres-exporter/vars/main.yml rename roles/custom/{matrix-prometheus-postgres-exporter/tasks => matrix-prometheus-services-proxy-connect/tasks/prometheus-postgres-exporter}/inject_into_nginx_proxy.yml (51%) diff --git a/CHANGELOG.md b/CHANGELOG.md index edb4dba9..bb62d8dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,18 @@ +# 2023-02-05 + +## The matrix-prometheus-postgres-exporter role lives independently now + +**TLDR**: the `matrix-prometheus-postgres-exporter` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +The `matrix-prometheus-postgres-exporter` role (which configures [Prometheus Postgres Exporter](https://github.com/prometheus-community/postgres_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter. + +It's still part of the playbook, but is now installed via `ansible-galaxy` (by running `just roles` / `make roles`). Some variables have been renamed (`matrix_prometheus_postgres_exporter_` -> `prometheus_postgres_exporter_`, etc.). The playbook will report all variables that you need to rename to get upgraded. All functionality remains intact. + +The `matrix-prometheus-services-proxy-connect` role has bee adjusted to help integrate the new `prometheus_postgres_exporter` role with our own services (`matrix-nginx-proxy`) + +Other roles which aren't strictly related to Matrix are likely to follow this fate of moving to their own repositories. Extracting them out allows other Ansible playbooks to make use of these roles easily. + + # 2023-01-26 ## Coturn can now use host-networking @@ -519,7 +534,7 @@ See our [Setting up the ntfy push notifications server](docs/configuring-playboo **If you're using node-exporter** (`matrix_prometheus_node_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_node_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/node-exporter`. -**If you're using [postgres-exporter](docs/configuring-playbook-prometheus-postgres.md)** (`matrix_prometheus_postgres_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_postgres_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/postgres-exporter`. +**If you're using [postgres-exporter](docs/configuring-playbook-prometheus-postgres.md)** (`prometheus_postgres_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/postgres-exporter`. **If you're using Synapse** and would like to collect its metrics from an external Prometheus server, you may find that: diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index 703c03b2..956b0573 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -13,7 +13,7 @@ matrix_prometheus_enabled: true prometheus_node_exporter_enabled: true # You can remove this, if unnecessary. -matrix_prometheus_postgres_exporter_enabled: true +prometheus_postgres_exporter_enabled: true # You can remove this, if unnecessary. matrix_prometheus_nginxlog_exporter_enabled: true @@ -41,7 +41,7 @@ Name | Description -----|---------- `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. `prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures -`matrix_prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. +`prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. `matrix_prometheus_nginxlog_exporter_enabled`|[NGINX Log Exporter](configuring-playbook-prometheus-nginxlog.md) is an addon of sorts to expose NGINX logs to Prometheus. `matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in `matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option. @@ -76,9 +76,9 @@ Name | Description `matrix_synapse_metrics_proxying_enabled`|Set this to `true` to expose Synapse's metrics on `https://matrix.DOMAIN/metrics/synapse/main-process` and `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`). Read [below](#collecting-synapse-worker-metrics-to-an-external-prometheus-server) if you're running a Synapse worker setup (`matrix_synapse_workers_enabled: true`). `prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network) `matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) -`matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network) +`prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network) `matrix_prometheus_nginxlog_exporter_enabled`|Set this to `true` to enable the [NGINX Log exporter](configuring-playbook-prometheus-nginxlog.md) (locally, on the container network) -`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) `matrix_bridge_hookshot_metrics_enabled`|Set this to `true` to make [Hookshot](configuring-playbook-bridge-hookshot.md) expose metrics (locally, on the container network) `matrix_bridge_hookshot_metrics_proxying_enabled`|Set this to `true` to expose the [Hookshot](configuring-playbook-bridge-hookshot.md) metrics on `https://matrix.DOMAIN/metrics/hookshot` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) `matrix_SERVICE_metrics_proxying_enabled`|Various other services/roles may provide similar `_metrics_enabled` and `_metrics_proxying_enabled` variables for exposing their metrics. Refer to each role for details. Only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` diff --git a/docs/configuring-playbook-prometheus-postgres.md b/docs/configuring-playbook-prometheus-postgres.md index 6fd13a9e..c3c8e0bf 100644 --- a/docs/configuring-playbook-prometheus-postgres.md +++ b/docs/configuring-playbook-prometheus-postgres.md @@ -6,17 +6,17 @@ You can enable this with the following settings in your configuration file (`inv ```yaml -matrix_prometheus_postgres_exporter_enabled: true +prometheus_postgres_exporter_enabled: true ``` ## What does it do? Name | Description -----|---------- -`matrix_prometheus_postgres_exporter_enabled`|Enable the postgres prometheus exporter. This sets up the docker container, connects it to the database and adds a 'job' to the prometheus config which tells prometheus about this new exporter. The default is 'false' -`matrix_prometheus_postgres_exporter_database_username`| The 'username' for the user that the exporter uses to connect to the database. The default is 'matrix_prometheus_postgres_exporter' -`matrix_prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database. By default, this is auto-generated by the playbook -`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|If set to `true`, exposes the Postgres exporter metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` for usage with an [external Prometheus server](configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`prometheus_postgres_exporter_enabled`|Enable the postgres prometheus exporter. This sets up the docker container, connects it to the database and adds a 'job' to the prometheus config which tells prometheus about this new exporter. The default is 'false' +`prometheus_postgres_exporter_database_username`| The 'username' for the user that the exporter uses to connect to the database. The default is 'matrix_prometheus_postgres_exporter' +`prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database. By default, this is auto-generated by the playbook +`matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled`|If set to `true`, exposes the Postgres exporter metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` for usage with an [external Prometheus server](configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) ## More information diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 2a8b2443..6d096dea 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -301,11 +301,11 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-prometheus.service', 'priority': 4000, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-core']}] if matrix_prometheus_enabled else []) + - ([{'name': (prometheus_node_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if prometheus_node_exporter_enabled else []) + ([{'name': (prometheus_node_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if prometheus_node_exporter_enabled else []) + - ([{'name': 'matrix-prometheus-postgres-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_postgres_exporter_enabled else []) + ([{'name': (prometheus_postgres_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if prometheus_postgres_exporter_enabled else []) + - ([{'name': 'matrix-prometheus-nginxlog-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_nginxlog_exporter_enabled else []) + ([{'name': 'matrix-prometheus-nginxlog-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if matrix_prometheus_nginxlog_exporter_enabled else []) + ([{'name': 'matrix-redis', 'priority': 750, 'groups': ['matrix', 'redis']}] if matrix_redis_enabled else []) + @@ -2589,10 +2589,10 @@ devture_postgres_managed_databases_auto: | }] if (matrix_etherpad_enabled and matrix_etherpad_database_engine == 'postgres' and matrix_etherpad_database_hostname == devture_postgres_connection_hostname) else []) + ([{ - 'name': matrix_prometheus_postgres_exporter_database_name, - 'username': matrix_prometheus_postgres_exporter_database_username, - 'password': matrix_prometheus_postgres_exporter_database_password, - }] if (matrix_prometheus_postgres_exporter_enabled and matrix_prometheus_postgres_exporter_database_hostname == devture_postgres_connection_hostname) else []) + 'name': prometheus_postgres_exporter_database_name, + 'username': prometheus_postgres_exporter_database_username, + 'password': prometheus_postgres_exporter_database_password, + }] if (prometheus_postgres_exporter_enabled and prometheus_postgres_exporter_database_hostname == devture_postgres_connection_hostname) else []) }} @@ -2976,6 +2976,45 @@ prometheus_node_exporter_container_labels_traefik_enabled: false # ###################################################################### +###################################################################### +# +# etke/prometheus_postgres_exporter +# +###################################################################### + +prometheus_postgres_exporter_enabled: false + +prometheus_postgres_exporter_identifier: matrix-prometheus-postgres-exporter + +prometheus_postgres_exporter_base_path: "{{ matrix_base_data_path }}/prometheus-postgres-exporter" + +prometheus_postgres_exporter_uid: "{{ matrix_user_uid }}" +prometheus_postgres_exporter_gid: "{{ matrix_user_gid }}" + +prometheus_postgres_exporter_server_fqn: "{{ matrix_server_fqn_matrix }}" + +prometheus_postgres_exporter_container_network: "{{ matrix_docker_network }}" + +prometheus_postgres_exporter_container_labels_traefik_enabled: false + +prometheus_postgres_exporter_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" +prometheus_postgres_exporter_database_username: matrix_prometheus_postgres_exporter +prometheus_postgres_exporter_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'prometheus.pg.db', rounds=655555) | to_uuid }}" +prometheus_postgres_exporter_database_name: matrix_prometheus_postgres_exporter + +prometheus_postgres_exporter_systemd_required_services_list: | + {{ + ['docker.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + }} + +###################################################################### +# +# /etke/prometheus_postgres_exporter +# +###################################################################### + ###################################################################### # # matrix-prometheus-nginxlog-exporter @@ -3011,8 +3050,8 @@ matrix_prometheus_scraper_synapse_rules_synapse_tag: "{{ matrix_synapse_docker_i matrix_prometheus_scraper_node_enabled: "{{ prometheus_node_exporter_enabled }}" matrix_prometheus_scraper_node_targets: "{{ [prometheus_node_exporter_identifier + ':9100'] if prometheus_node_exporter_enabled else [] }}" -matrix_prometheus_scraper_postgres_enabled: "{{ matrix_prometheus_postgres_exporter_enabled }}" -matrix_prometheus_scraper_postgres_targets: "{{ ['matrix-prometheus-postgres-exporter:'+ matrix_prometheus_postgres_exporter_port | string] if matrix_prometheus_scraper_postgres_enabled else [] }}" +matrix_prometheus_scraper_postgres_enabled: "{{ prometheus_postgres_exporter_enabled }}" +matrix_prometheus_scraper_postgres_targets: "{{ [prometheus_postgres_exporter_identifier + ':' + prometheus_postgres_exporter_port | string] if matrix_prometheus_scraper_postgres_enabled else [] }}" matrix_prometheus_scraper_hookshot_enabled: "{{ matrix_hookshot_metrics_enabled|default(false) }}" matrix_prometheus_scraper_hookshot_targets: "{{ [matrix_hookshot_container_url | string +':'+ matrix_hookshot_metrics_port | string] if matrix_hookshot_metrics_enabled else [] }}" @@ -3027,29 +3066,6 @@ matrix_prometheus_scraper_nginxlog_server_port: "{{ (matrix_prometheus_nginxlog_ # ###################################################################### -###################################################################### -# -# matrix-prometheus-postgres-exporter -# -###################################################################### - -matrix_prometheus_postgres_exporter_enabled: false -matrix_prometheus_postgres_exporter_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" -matrix_prometheus_postgres_exporter_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'prometheus.pg.db', rounds=655555) | to_uuid }}" - -matrix_prometheus_postgres_exporter_systemd_required_services_list: | - {{ - ['docker.service'] - + - ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) - }} - -###################################################################### -# -# /matrix-prometheus-postgres-exporter -# -###################################################################### - ###################################################################### # # matrix-grafana @@ -3069,7 +3085,7 @@ matrix_grafana_dashboard_download_urls: | + (prometheus_node_exporter_dashboard_urls if prometheus_node_exporter_enabled else []) + - (matrix_prometheus_postgres_exporter_dashboard_urls if matrix_prometheus_postgres_exporter_enabled else []) + (prometheus_postgres_exporter_dashboard_urls if prometheus_postgres_exporter_enabled else []) + (matrix_prometheus_nginxlog_exporter_dashboard_urls if matrix_prometheus_nginxlog_exporter_enabled else []) }} @@ -3083,13 +3099,6 @@ matrix_grafana_default_home_dashboard_path: |- }[matrix_homeserver_implementation] }} -matrix_grafana_systemd_wanted_services_list: | - {{ - [] - + - (['matrix-prometheus-postgres-exporter.service'] if matrix_prometheus_postgres_exporter_enabled else []) - }} - ###################################################################### # # /matrix-grafana diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 5190521e..7be0d629 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -87,7 +87,7 @@ - custom/matrix-conduit - custom/matrix-synapse-admin - galaxy/prometheus_node_exporter - - custom/matrix-prometheus-postgres-exporter + - galaxy/prometheus_postgres_exporter - custom/matrix-prometheus-nginxlog-exporter - custom/matrix-prometheus - custom/matrix-grafana diff --git a/requirements.yml b/requirements.yml index 8ffa99d5..69b09cac 100644 --- a/requirements.yml +++ b/requirements.yml @@ -32,3 +32,6 @@ - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git version: v1.5.0-1 + +- src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git + version: v0.11.1-0 diff --git a/roles/custom/matrix-prometheus-postgres-exporter/defaults/main.yml b/roles/custom/matrix-prometheus-postgres-exporter/defaults/main.yml deleted file mode 100644 index 78481a32..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/defaults/main.yml +++ /dev/null @@ -1,65 +0,0 @@ ---- -# matrix-prometheus-postgres-exporter is an Prometheus exporter for postgres metrics -# Project source code URL: https://github.com/prometheus-community/postgres_exporter - -matrix_prometheus_postgres_exporter_enabled: false - -matrix_prometheus_postgres_exporter_version: v0.11.1 -matrix_prometheus_postgres_exporter_port: 9187 - -matrix_prometheus_postgres_exporter_docker_image: "{{ matrix_prometheus_postgres_exporter_docker_image_name_prefix }}prometheuscommunity/postgres-exporter:{{ matrix_prometheus_postgres_exporter_version }}" -matrix_prometheus_postgres_exporter_docker_image_name_prefix: "quay.io/" -matrix_prometheus_postgres_exporter_docker_image_force_pull: "{{ matrix_prometheus_postgres_exporter_docker_image.endswith(':latest') }}" - -# A list of extra arguments to pass to the container -matrix_prometheus_postgres_exporter_container_extra_arguments: ["-e PG_EXPORTER_AUTO_DISCOVER_DATABASES=true", - "-e PG_EXPORTER_WEB_LISTEN_ADDRESS=\":{{ matrix_prometheus_postgres_exporter_port }}\"", - "-e DATA_SOURCE_NAME=\"postgresql://{{ matrix_prometheus_postgres_exporter_database_username }}:{{ matrix_prometheus_postgres_exporter_database_password }}@{{ matrix_prometheus_postgres_exporter_database_hostname }}:5432/{{ matrix_prometheus_postgres_exporter_database_name }}?sslmode=disable\""] - -# List of systemd services that matrix-prometheus-postgres-exporter.service depends on -matrix_prometheus_postgres_exporter_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-prometheus-postgres-exporter.service wants -matrix_prometheus_postgres_exporter_systemd_wanted_services_list: [] - -# details for connecting to the database -matrix_prometheus_postgres_exporter_database_username: 'matrix_prometheus_postgres_exporter' -matrix_prometheus_postgres_exporter_database_password: 'some-password' -matrix_prometheus_postgres_exporter_database_hostname: '' -matrix_prometheus_postgres_exporter_database_port: 5432 -matrix_prometheus_postgres_exporter_database_name: 'matrix_prometheus_postgres_exporter' - -# Controls whether postgres-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/postgres-exporter`. -# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. -# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. -matrix_prometheus_postgres_exporter_metrics_proxying_enabled: false - -# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9187 in the container). -# -# Takes an ":" value (e.g. "127.0.0.1:9187"), just a port number or an empty string to not expose. -# -# You likely don't need to do this. See `matrix_prometheus_postgres_exporter_metrics_proxying_enabled`. -# -# Official recommendations are to run this container with `--net=host`, -# but we don't do that, since it: -# - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008) -# - or listens on a loopback interface only (--net=host and 127.0.0.1:9187), which is not reachable from another container (like `matrix-prometheus`) -# -# Using `--net=host` and binding to Docker's `matrix` bridge network may be a solution to both, -# but that's trickier to accomplish and won't necessarily work (hasn't been tested). -# -# Not using `--net=host` means that our network statistic reports are likely broken (inaccurate), -# because node-exporter can't see all interfaces, etc. -# For now, we'll live with that, until someone develops a better solution. -matrix_prometheus_postgres_exporter_container_http_host_bind_port: '' - -# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. -# If matrix_prometheus_postgres_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" -# If matrix_prometheus_postgres_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used -# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) -matrix_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + matrix_prometheus_postgres_exporter_container_http_host_bind_port_number_raw if not ':' in matrix_prometheus_postgres_exporter_container_http_host_bind_port else (matrix_prometheus_postgres_exporter_container_http_host_bind_port if matrix_prometheus_postgres_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" - -# matrix_prometheus_postgres_exporter_dashboard_urls contains a list of URLs with Grafana dashboard definitions. -# If the Grafana role is enabled, these dashboards will be downloaded. -matrix_prometheus_postgres_exporter_dashboard_urls: - - "https://grafana.com/api/dashboards/9628/revisions/7/download" diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/main.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/main.yml deleted file mode 100644 index 7ff6d15a..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/main.yml +++ /dev/null @@ -1,29 +0,0 @@ ---- - -- block: - - when: matrix_prometheus_postgres_exporter_enabled | bool and matrix_prometheus_postgres_exporter_metrics_proxying_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: - - setup-all - - setup-nginx-proxy - - install-all - - install-nginx-proxy - -- block: - - when: matrix_prometheus_postgres_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - - - when: matrix_prometheus_postgres_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-prometheus-postgres-exporter - - install-all - - install-prometheus-postgres-exporter - -- block: - - when: not matrix_prometheus_postgres_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-prometheus-postgres-exporter diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_install.yml deleted file mode 100644 index ee0f9bef..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_install.yml +++ /dev/null @@ -1,19 +0,0 @@ ---- - -- name: Ensure matrix-prometheus-postgres-exporter image is pulled - community.docker.docker_image: - name: "{{ matrix_prometheus_postgres_exporter_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_prometheus_postgres_exporter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_postgres_exporter_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure matrix-prometheus-postgres-exporter.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-prometheus-postgres-exporter.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-postgres-exporter.service" - mode: 0644 - register: matrix_prometheus_postgres_exporter_systemd_service_result diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_uninstall.yml deleted file mode 100644 index bdf72042..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_uninstall.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- name: Check existence of matrix-prometheus-postgres-exporter service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-postgres-exporter.service" - register: matrix_prometheus_postgres_exporter_service_stat - -- when: matrix_prometheus_postgres_exporter_service_stat.stat.exists | bool - block: - - name: Ensure matrix-prometheus-postgres-exporter is stopped - ansible.builtin.service: - name: matrix-prometheus-postgres-exporter - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-prometheus-postgres-exporter.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-postgres-exporter.service" - state: absent diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/validate_config.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/validate_config.yml deleted file mode 100644 index 6acf784f..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/validate_config.yml +++ /dev/null @@ -1,9 +0,0 @@ ---- - -- name: Fail if required prometheus-postgres-exporter settings not defined - ansible.builtin.fail: - msg: > - You need to define a required configuration setting (`{{ item }}`). - when: "vars[item] == ''" - with_items: - - matrix_prometheus_postgres_exporter_database_hostname diff --git a/roles/custom/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2 b/roles/custom/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2 deleted file mode 100644 index 5c3fbac6..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2 +++ /dev/null @@ -1,42 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=matrix-prometheus-postgres-exporter -{% for service in matrix_prometheus_postgres_exporter_systemd_required_services_list %} -Requires={{ service }} -After={{ service }} -{% endfor %} -{% for service in matrix_prometheus_postgres_exporter_systemd_wanted_services_list %} -Wants={{ service }} -{% endfor %} -DefaultDependencies=no - -[Service] -Type=simple -Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-postgres-exporter 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-postgres-exporter 2>/dev/null || true' - - -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-prometheus-postgres-exporter \ - --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --cap-drop=ALL \ - --read-only \ - {% for arg in matrix_prometheus_postgres_exporter_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - --network={{ matrix_docker_network }} \ - {% if matrix_prometheus_postgres_exporter_container_http_host_bind_port %} - -p {{ matrix_prometheus_postgres_exporter_container_http_host_bind_port }}:{{matrix_prometheus_postgres_exporter_port}} \ - {% endif %} - --pid=host \ - {{ matrix_prometheus_postgres_exporter_docker_image }} - -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-postgres-exporter 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-postgres-exporter 2>/dev/null || true' -Restart=always -RestartSec=30 -SyslogIdentifier=matrix-prometheus-postgres-exporter - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix-prometheus-postgres-exporter/vars/main.yml b/roles/custom/matrix-prometheus-postgres-exporter/vars/main.yml deleted file mode 100644 index aed3b216..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/vars/main.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- - -# `matrix_prometheus_postgres_exporter_container_http_host_bind_port_number_raw` contains the raw port number extracted from `matrix_prometheus_postgres_exporter_container_http_host_bind_port`, -# which can contain values like this: ('1234', '127.0.0.1:1234', '0.0.0.0:1234') -matrix_prometheus_postgres_exporter_container_http_host_bind_port_number_raw: "{{ '' if matrix_prometheus_postgres_exporter_container_http_host_bind_port == '' else (matrix_prometheus_postgres_exporter_container_http_host_bind_port.split(':')[1] if ':' in matrix_prometheus_postgres_exporter_container_http_host_bind_port else matrix_prometheus_postgres_exporter_container_http_host_bind_port) }}" diff --git a/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml b/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml index 5799d053..30ef2c13 100644 --- a/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml +++ b/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml @@ -13,3 +13,15 @@ matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxyi # If prometheus_node_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used # Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) matrix_prometheus_services_proxy_connect_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + prometheus_node_exporter_container_http_host_bind_port_number_raw if not ':' in prometheus_node_exporter_container_http_host_bind_port else (prometheus_node_exporter_container_http_host_bind_port if prometheus_node_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" + + +# Controls whether postgres-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/postgres-exporter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled: false + +# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. +# If prometheus_node_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" +# If prometheus_node_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used +# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) +matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + prometheus_postgres_exporter_container_http_host_bind_port_number_raw if not ':' in prometheus_postgres_exporter_container_http_host_bind_port else (prometheus_postgres_exporter_container_http_host_bind_port if prometheus_postgres_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" diff --git a/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml index 7f35a318..e5654688 100644 --- a/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml @@ -17,3 +17,12 @@ - setup-nginx-proxy - install-all - install-nginx-proxy + +- block: + - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml similarity index 51% rename from roles/custom/matrix-prometheus-postgres-exporter/tasks/inject_into_nginx_proxy.yml rename to roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml index 121efc17..34796081 100644 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml @@ -1,25 +1,16 @@ --- -- name: Fail if matrix-nginx-proxy role already executed - ansible.builtin.fail: - msg: >- - Trying to append postgres-exporter's reverse-proxying configuration to matrix-nginx-proxy, - but it's pointless since the matrix-nginx-proxy role had already executed. - To fix this, please change the order of roles in your playbook, - so that the matrix-nginx-proxy role would run after the matrix-prometheus-postgres-exporter role. - when: matrix_nginx_proxy_role_executed | default(False) | bool - - name: Generate postgres-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/postgres-exporter) ansible.builtin.set_fact: - matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block: | + matrix_prometheus_services_proxy_connect_postgres_exporter_nginx_metrics_configuration_block: | location /metrics/postgres-exporter { {% if matrix_nginx_proxy_enabled | default(False) %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; - set $backend "matrix-prometheus-postgres-exporter:9187"; + set $backend "{{ prometheus_postgres_exporter_identifier }}:{{ prometheus_postgres_exporter_port }}"; proxy_pass http://$backend/metrics; - {% elif matrix_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} - proxy_pass http://{{ matrix_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; + {% elif matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} + proxy_pass http://{{ matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; {% else %} return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; {% endif %} @@ -31,5 +22,5 @@ {{ matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + - [matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block] + [matrix_prometheus_services_proxy_connect_postgres_exporter_nginx_metrics_configuration_block] }} diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml index b553160a..496c4c75 100644 --- a/roles/custom/matrix_playbook_migration/defaults/main.yml +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -2,3 +2,6 @@ # Controls if (`matrix_prometheus_node_exporter` -> `prometheus_node_exporter`) validation will run. matrix_playbook_migration_matrix_prometheus_node_exporter_migration_validation_enabled: true + +# Controls if (`matrix_prometheus_postgres_exporter` -> `prometheus_postgres_exporter`) validation will run. +matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 5d22e4bf..b04c9e0e 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -35,6 +35,7 @@ - {'old': 'matrix_common_after_systemd_service_start_wait_for_timeout_seconds', 'new': 'devture_systemd_service_manager_up_verification_delay_seconds'} - {'old': 'matrix_systemd_services_autostart_enabled', 'new': 'devture_systemd_service_manager_services_autostart_enabled'} - {'old': 'matrix_prometheus_node_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled'} + - {'old': 'matrix_prometheus_postgres_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled'} - name: (Deprecation) Catch and report matrix_postgres variables ansible.builtin.fail: @@ -64,3 +65,18 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_prometheus_node_exporter_` -> `prometheus_node_exporter_`). We found usage of the following variables: {{ matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_prometheus_postgres_exporter_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_prometheus_postgres_exporter variables + ansible.builtin.fail: + msg: >- + The matrix-prometheus-postgres-exporter role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_prometheus_postgres_exporter_` -> `prometheus_postgres_exporter_`). + We found usage of the following variables: {{ matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars | length > 0" From 045ed94d43fbaa63befc3f2a863311b86b02c56c Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 5 Feb 2023 10:54:51 +0200 Subject: [PATCH 047/382] Upgrade prometheus_postgres_exporter (v0.11.1-0 -> v0.11.1-1) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 69b09cac..02521cbc 100644 --- a/requirements.yml +++ b/requirements.yml @@ -34,4 +34,4 @@ version: v1.5.0-1 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git - version: v0.11.1-0 + version: v0.11.1-1 From e018663ba4f2721f58dacad58a23da9b776c5fff Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Feb 2023 08:38:43 +0200 Subject: [PATCH 048/382] Attach ma1sd/nginx-proxy/synapse-reverse-proxy-companion to additional networks in a better way Switching from doing "post-start" loop hacks to running the container in 3 steps: `create` + potentially connect to additional networks + `start`. This way, the container would be connected to all its networks even at the very beginning of its life. --- .../templates/systemd/matrix-ma1sd.service.j2 | 8 ++++++-- .../templates/systemd/matrix-nginx-proxy.service.j2 | 8 ++++++-- .../matrix-synapse-reverse-proxy-companion.service.j2 | 6 ++++-- 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 b/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 index 19bd5720..55505ab6 100644 --- a/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 +++ b/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 @@ -18,7 +18,9 @@ ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_s # ma1sd writes an SQLite shared library (libsqlitejdbc.so) to /tmp and executes it from there, # so /tmp needs to be mounted with an exec option. -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-ma1sd \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-ma1sd \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ @@ -39,9 +41,11 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_ma1sd_docker_image }} {% for network in matrix_ma1sd_container_additional_networks %} -ExecStartPost={{ devture_systemd_docker_base_host_command_sh }} -c 'container_name=matrix-ma1sd; network_name={{ network }}; attempt=0; while [ $attempt -le 29 ]; do attempt=$(( $attempt + 1 )); if [ "`{{ devture_systemd_docker_base_host_command_docker }} inspect -f {{ '{{.State.Running}}' }} $container_name 2> /dev/null`" = "true" ]; then break; fi; sleep 1; done; {{ devture_systemd_docker_base_host_command_docker }} network connect $network_name $container_name' +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-ma1sd {% endfor %} +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-ma1sd + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-ma1sd 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-ma1sd 2>/dev/null || true' Restart=always diff --git a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 index 0a7e9052..e6c94132 100755 --- a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 @@ -16,7 +16,9 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-nginx-proxy 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-nginx-proxy 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-nginx-proxy \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-nginx-proxy \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ @@ -48,9 +50,11 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_nginx_proxy_docker_image }} {% for network in matrix_nginx_proxy_container_additional_networks %} -ExecStartPost={{ devture_systemd_docker_base_host_command_sh }} -c 'container_name=matrix-nginx-proxy; network_name={{ network }}; attempt=0; while [ $attempt -le 29 ]; do attempt=$(( $attempt + 1 )); if [ "`{{ devture_systemd_docker_base_host_command_docker }} inspect -f {{ '{{.State.Running}}' }} $container_name 2> /dev/null`" = "true" ]; then break; fi; sleep 1; done; {{ devture_systemd_docker_base_host_command_docker }} network connect $network_name $container_name' +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-nginx-proxy {% endfor %} +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-nginx-proxy + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-nginx-proxy 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-nginx-proxy 2>/dev/null || true' ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-nginx-proxy /usr/sbin/nginx -s reload diff --git a/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 b/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 index 2b548ef8..125d7f49 100755 --- a/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 +++ b/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 @@ -16,7 +16,7 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-reverse-proxy-companion 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-reverse-proxy-companion 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --rm \ --name=matrix-synapse-reverse-proxy-companion \ --log-driver=none \ @@ -39,9 +39,11 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run \ {{ matrix_synapse_reverse_proxy_companion_container_image }} {% for network in matrix_synapse_reverse_proxy_companion_container_additional_networks %} -ExecStartPost={{ devture_systemd_docker_base_host_command_sh }} -c 'container_name=matrix-synapse-reverse-proxy-companion; network_name={{ network }}; attempt=0; while [ $attempt -le 29 ]; do attempt=$(( $attempt + 1 )); if [ "`{{ devture_systemd_docker_base_host_command_docker }} inspect -f {{ '{{.State.Running}}' }} $container_name 2> /dev/null`" = "true" ]; then break; fi; sleep 1; done; {{ devture_systemd_docker_base_host_command_docker }} network connect $network_name $container_name' +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-synapse-reverse-proxy-companion {% endfor %} +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-synapse-reverse-proxy-companion + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-reverse-proxy-companion 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-reverse-proxy-companion 2>/dev/null || true' ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-synapse-reverse-proxy-companion /usr/sbin/nginx -s reload From 4d6a8d049d7436a8c487661f67f57c6b92da0e21 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Feb 2023 08:48:11 +0200 Subject: [PATCH 049/382] Add matrix_nginx_proxy_container_network variable --- roles/custom/matrix-nginx-proxy/defaults/main.yml | 4 ++++ .../templates/bin/lets-encrypt-certificates-renew.j2 | 2 +- .../templates/systemd/matrix-nginx-proxy.service.j2 | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 6fbc019e..2861084d 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -21,6 +21,10 @@ matrix_nginx_proxy_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-nginx-proxy.service wants matrix_nginx_proxy_systemd_wanted_services_list: [] +# The base container network. +# Also see: matrix_nginx_proxy_container_additional_networks +matrix_nginx_proxy_container_network: "{{ matrix_docker_network }}" + # A list of additional container networks that matrix-nginx-proxy would be connected to. # The playbook does not create these networks, so make sure they already exist. # diff --git a/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 b/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 index 89113629..5f235ea2 100644 --- a/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 @@ -12,7 +12,7 @@ docker run \ --name=matrix-certbot \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ - --network="{{ matrix_docker_network }}" \ + --network="{{ matrix_nginx_proxy_container_network }}" \ -p 127.0.0.1:{{ matrix_ssl_lets_encrypt_certbot_standalone_http_port }}:8080 \ --mount type=bind,src={{ matrix_ssl_config_dir_path }},dst=/etc/letsencrypt \ --mount type=bind,src={{ matrix_ssl_log_dir_path }},dst=/var/log/letsencrypt \ diff --git a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 index e6c94132..301eb9d5 100755 --- a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 @@ -24,7 +24,7 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --cap-drop=ALL \ --read-only \ --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_nginx_proxy_tmp_directory_size_mb }}m \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_nginx_proxy_container_network }} \ {% if matrix_nginx_proxy_container_http_host_bind_port %} -p {{ matrix_nginx_proxy_container_http_host_bind_port }}:8080 \ {% endif %} From 5de5b5c62c5efe9e71e225e673c72932177f95f9 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Feb 2023 09:49:15 +0200 Subject: [PATCH 050/382] Upgrade prometheus_postgres_exporter (v0.11.1-1 -> v0.11.1-2) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 02521cbc..d580dbde 100644 --- a/requirements.yml +++ b/requirements.yml @@ -34,4 +34,4 @@ version: v1.5.0-1 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git - version: v0.11.1-1 + version: v0.11.1-2 From 946bbe9734a0b045285e738d5ab52a44d061ab71 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Feb 2023 09:54:34 +0200 Subject: [PATCH 051/382] Upgrade prometheus_node_exporter (v1.5.0-1 -> v1.5.0-2) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index d580dbde..4797423d 100644 --- a/requirements.yml +++ b/requirements.yml @@ -31,7 +31,7 @@ version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git - version: v1.5.0-1 + version: v1.5.0-2 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.11.1-2 From f9836046956bb7fb3741dcafb2e37a55047b67b5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Feb 2023 10:34:51 +0200 Subject: [PATCH 052/382] Initial work on Traefik support This gets us started on adding a Traefik role and hooking Traefik: - directly to services which support Traefik - we only have a few of these right now, but the list will grow - to matrix-nginx-proxy for most services that integrate with matrix-nginx-proxy right now Traefik usage should be disabled by default for now and nothing should change for people just yet. Enabling these experiments requires additional configuration like this: ```yaml devture_traefik_ssl_email_address: '.....' matrix_playbook_traefik_role_enabled: true matrix_playbook_traefik_labels_enabled: true matrix_ssl_retrieval_method: none matrix_nginx_proxy_https_enabled: false matrix_nginx_proxy_container_http_host_bind_port: '' matrix_nginx_proxy_container_federation_host_bind_port: '' matrix_nginx_proxy_trust_forwarded_proto: true matrix_nginx_proxy_x_forwarded_for: '$proxy_add_x_forwarded_for' matrix_coturn_enabled: false ``` What currently works is: reverse-proxying for all nginx-proxy based services **except** for the Matrix homeserver (both Client-Server an Federation traffic for the homeserver don't work yet) --- group_vars/matrix_servers | 70 +++++++++ playbooks/matrix.yml | 3 + requirements.yml | 3 + .../matrix-nginx-proxy/defaults/main.yml | 74 +++++++++ .../tasks/setup_nginx_proxy.yml | 8 + .../matrix-nginx-proxy/templates/labels.j2 | 144 ++++++++++++++++++ .../systemd/matrix-nginx-proxy.service.j2 | 1 + 7 files changed, 303 insertions(+) create mode 100644 roles/custom/matrix-nginx-proxy/templates/labels.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6d096dea..da624d9a 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -19,6 +19,28 @@ # Also see `devture_docker_sdk_for_python_installation_enabled`. matrix_playbook_docker_installation_enabled: true +# Controls whether to run the Traefik role or not +# See the `com.devture.ansible.role.traefik` section below for role configuration. +# +# There's a difference between `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`. +# `devture_traefik_enabled` controls what the Traefik role would do - when not enabled, it will run uninstall tasks, etc. +# `matrix_playbook_traefik_role_enabled` controls if the Traefik role would even run at all. +# +# Sometimes, you're installing Traefik via a different (related playbook) which uses the same role. +# In such cases, you'd like to disable the role in this playbook from bothering with Traefik at all (`matrix_playbook_traefik_role_enabled: false`). +# If you used `devture_traefik_enabled: false` + `matrix_playbook_traefik_role_enabled: true` instead, you'd see the Treafik role here +# try to delete Traefik data (`/devture-traefik`) installed by the other playbook. +matrix_playbook_traefik_role_enabled: false + +# Controls whether to attach Traefik labels to services. +# This is separate from `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`, +# because you may wish to disable Traefik installation by the playbook, yet still use Traefik +# installed in another way. +matrix_playbook_traefik_labels_enabled: false + +# Controls the additional network that reverse-proxyable services will be connected to. +matrix_playbook_reverse_proxyable_services_additional_network: "{{ devture_traefik_container_network if devture_traefik_enabled else '' }}" + ######################################################################## # # # /Playbook # @@ -320,6 +342,8 @@ devture_systemd_service_manager_services_list_auto: | ([{'name': 'matrix-synapse-admin.service', 'priority': 4000, 'groups': ['matrix', 'synapse-admin']}] if matrix_synapse_admin_enabled else []) + ([{'name': 'matrix-synapse-reverse-proxy-companion.service', 'priority': 1500, 'groups': ['matrix', 'homeservers', 'synapse', 'reverse-proxies']}] if matrix_synapse_reverse_proxy_companion_enabled else []) + + + ([{'name': 'devture-traefik.service', 'priority': 3000, 'groups': ['matrix', 'traefik', 'reverse-proxies']}] if matrix_playbook_traefik_role_enabled else []) }} ######################################################################## @@ -2157,6 +2181,8 @@ matrix_ma1sd_database_password: "{{ '%s' | format(matrix_homeserver_generic_secr # If that's not the case, you may wish to disable this and take care of proxying yourself. matrix_nginx_proxy_enabled: true +matrix_nginx_proxy_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "{{ 'matrix-corporal:41080' if matrix_corporal_enabled else 'matrix-nginx-proxy:12080' }}" matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "{{ '127.0.0.1:41080' if matrix_corporal_enabled else '127.0.0.1:12080' }}" matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb: |- @@ -2185,6 +2211,22 @@ matrix_nginx_proxy_proxy_grafana_enabled: "{{ matrix_grafana_enabled }}" matrix_nginx_proxy_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" matrix_nginx_proxy_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" +matrix_nginx_proxy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_config_entrypoint_primary }}" + +matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled: "{{ matrix_client_element_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_grafana_enabled: "{{ matrix_grafana_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" + matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: "{{ matrix_corporal_enabled and matrix_corporal_http_api_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corporal:41081" matrix_nginx_proxy_proxy_matrix_corporal_api_addr_sans_container: "127.0.0.1:41081" @@ -2968,7 +3010,11 @@ prometheus_node_exporter_server_fqn: "{{ matrix_server_fqn_matrix }}" prometheus_node_exporter_container_network: "{{ matrix_docker_network }}" +prometheus_node_exporter_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + prometheus_node_exporter_container_labels_traefik_enabled: false +prometheus_node_exporter_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +prometheus_node_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_config_entrypoint_primary }}" ###################################################################### # @@ -2995,7 +3041,11 @@ prometheus_postgres_exporter_server_fqn: "{{ matrix_server_fqn_matrix }}" prometheus_postgres_exporter_container_network: "{{ matrix_docker_network }}" +prometheus_postgres_exporter_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + prometheus_postgres_exporter_container_labels_traefik_enabled: false +prometheus_postgres_exporter_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +prometheus_postgres_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_config_entrypoint_primary }}" prometheus_postgres_exporter_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" prometheus_postgres_exporter_database_username: matrix_prometheus_postgres_exporter @@ -3284,3 +3334,23 @@ matrix_user_creator_users_auto: | # /matrix-user-creator # ###################################################################### + + +######################################################################## +# # +# com.devture.ansible.role.traefik # +# # +######################################################################## + +# To completely disable the Traefik role from running, use `matrix_playbook_traefik_role_enabled: false`. +# See the comment there for more details about why we have both `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`. +devture_traefik_enabled: "{{ matrix_playbook_traefik_role_enabled }}" + +devture_traefik_uid: "{{ matrix_user_uid }}" +devture_traefik_gid: "{{ matrix_user_gid }}" + +######################################################################## +# # +# /com.devture.ansible.role.traefik # +# # +######################################################################## diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 7be0d629..392a3a0e 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -115,6 +115,9 @@ - custom/matrix-user-creator - custom/matrix-common-after + - when: matrix_playbook_traefik_role_enabled | bool + role: galaxy/com.devture.ansible.role.traefik + - when: devture_systemd_service_manager_enabled | bool role: galaxy/com.devture.ansible.role.systemd_service_manager diff --git a/requirements.yml b/requirements.yml index 4797423d..696b2700 100644 --- a/requirements.yml +++ b/requirements.yml @@ -35,3 +35,6 @@ - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.11.1-2 + +- src: git+https://github.com/devture/com.devture.ansible.role.traefik.git + version: 407af71a3667b1d8083beb10bf22423ecf013f58 diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 2861084d..25911a49 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -40,6 +40,80 @@ matrix_nginx_proxy_container_additional_networks: [] # Contains definition objects like this: `{"src": "/outside", "dst": "/inside", "options": "rw|ro|slave|.."} matrix_nginx_proxy_container_additional_volumes: [] +# matrix_nginx_proxy_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_nginx_proxy_container_labels_additional_labels`. +matrix_nginx_proxy_container_labels_traefik_enabled: false +matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_nginx_proxy_container_network }}" +matrix_nginx_proxy_container_labels_traefik_entrypoints: web-secure +matrix_nginx_proxy_container_labels_traefik_tls_certResolver: default # noqa var-naming + +matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_element_hostname: "{{ matrix_server_fqn_element }}" +matrix_nginx_proxy_container_labels_traefik_proxy_element_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_element_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_element_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_hostname: "{{ matrix_server_fqn_hydrogen }}" +matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_cinny_hostname: "{{ matrix_server_fqn_cinny }}" +matrix_nginx_proxy_container_labels_traefik_proxy_cinny_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_cinny_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_cinny_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_hostname: "{{ matrix_server_fqn_buscarron }}" +matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname: "{{ matrix_server_fqn_dimension }}" +matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_dimension_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_hostname: "{{ matrix_server_fqn_etherpad }}" +matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_hostname: "{{ matrix_server_fqn_bot_go_neb }}" +matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_hostname: "{{ matrix_server_fqn_jitsi }}" +matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_grafana_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_grafana_hostname: "{{ matrix_server_fqn_grafana }}" +matrix_nginx_proxy_container_labels_traefik_proxy_grafana_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_grafana_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_grafana_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}" +matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_hostname }}`)" + +matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_hostname: "{{ matrix_server_fqn_ntfy }}" +matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_hostname }}`)" + +# matrix_nginx_proxy_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_nginx_proxy_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_nginx_proxy_container_labels_additional_labels: '' + + # A list of extra arguments to pass to the container matrix_nginx_proxy_container_extra_arguments: [] diff --git a/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml b/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml index dd11721a..31ff68f2 100644 --- a/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml +++ b/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml @@ -22,6 +22,14 @@ - "{{ matrix_nginx_proxy_data_path }}" - "{{ matrix_nginx_proxy_confd_path }}" +- name: Ensure Matrix nginx-proxy labels file is created + ansible.builtin.template: + src: "{{ role_path }}/templates/labels.j2" + dest: "{{ matrix_nginx_proxy_base_path }}/labels" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + mode: 0640 + - name: Ensure Matrix nginx-proxy configured (main config override) ansible.builtin.template: src: "{{ role_path }}/templates/nginx/nginx.conf.j2" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 new file mode 100644 index 00000000..06cc8d1a --- /dev/null +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -0,0 +1,144 @@ +{% if matrix_nginx_proxy_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_nginx_proxy_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_nginx_proxy_container_labels_traefik_docker_network }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled %} +# Element +traefik.http.routers.matrix-nginx-proxy-element.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_element_rule }} +traefik.http.routers.matrix-nginx-proxy-element.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-element.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_element_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_element_tls %} +traefik.http.routers.matrix-nginx-proxy-element.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-element.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled %} +# Hydrogen +traefik.http.routers.matrix-nginx-proxy-hydrogen.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_rule }} +traefik.http.routers.matrix-nginx-proxy-hydrogen.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-hydrogen.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_tls %} +traefik.http.routers.matrix-nginx-proxy-hydrogen.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-hydrogen.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled %} +# Cinny +traefik.http.routers.matrix-nginx-proxy-cinny.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_cinny_rule }} +traefik.http.routers.matrix-nginx-proxy-cinny.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-cinny.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_cinny_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_cinny_tls %} +traefik.http.routers.matrix-nginx-proxy-cinny.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-cinny.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled %} +# Buscarron +traefik.http.routers.matrix-nginx-proxy-buscarron.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_rule }} +traefik.http.routers.matrix-nginx-proxy-buscarron.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-buscarron.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_tls %} +traefik.http.routers.matrix-nginx-proxy-buscarron.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-buscarron.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled %} +# Dimension +traefik.http.routers.matrix-nginx-proxy-dimension.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_rule }} +traefik.http.routers.matrix-nginx-proxy-dimension.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-dimension.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls %} +traefik.http.routers.matrix-nginx-proxy-dimension.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-dimension.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled %} +# Etherpad +traefik.http.routers.matrix-nginx-proxy-etherpad.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_rule }} +traefik.http.routers.matrix-nginx-proxy-etherpad.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-etherpad.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls %} +traefik.http.routers.matrix-nginx-proxy-etherpad.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-etherpad.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled %} +# Go NEB bot +traefik.http.routers.matrix-nginx-proxy-bot_go_neb.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_rule }} +traefik.http.routers.matrix-nginx-proxy-bot_go_neb.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-bot_go_neb.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_tls %} +traefik.http.routers.matrix-nginx-proxy-bot_go_neb.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-bot_go_neb.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled %} +# Jitsi +traefik.http.routers.matrix-nginx-proxy-jitsi.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_rule }} +traefik.http.routers.matrix-nginx-proxy-jitsi.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-jitsi.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_tls %} +traefik.http.routers.matrix-nginx-proxy-jitsi.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-jitsi.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_grafana_enabled %} +# Grafana +traefik.http.routers.matrix-nginx-proxy-grafana.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_grafana_rule }} +traefik.http.routers.matrix-nginx-proxy-grafana.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-grafana.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_grafana_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_grafana_tls %} +traefik.http.routers.matrix-nginx-proxy-grafana.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-grafana.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled %} +# Sygnal +traefik.http.routers.matrix-nginx-proxy-sygnal.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_rule }} +traefik.http.routers.matrix-nginx-proxy-sygnal.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-sygnal.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls %} +traefik.http.routers.matrix-nginx-proxy-sygnal.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-sygnal.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +{% if matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_enabled %} +# ntfy +traefik.http.routers.matrix-nginx-proxy-ntfy.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_rule }} +traefik.http.routers.matrix-nginx-proxy-ntfy.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-ntfy.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_tls %} +traefik.http.routers.matrix-nginx-proxy-ntfy.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-ntfy.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + +traefik.http.services.matrix-nginx-proxy-web.loadbalancer.server.port=8080 +{% endif %} + +{{ matrix_nginx_proxy_container_labels_additional_labels }} diff --git a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 index 301eb9d5..f302c92e 100755 --- a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 @@ -24,6 +24,7 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --cap-drop=ALL \ --read-only \ --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_nginx_proxy_tmp_directory_size_mb }}m \ + --label-file={{ matrix_nginx_proxy_base_path }}/labels \ --network={{ matrix_nginx_proxy_container_network }} \ {% if matrix_nginx_proxy_container_http_host_bind_port %} -p {{ matrix_nginx_proxy_container_http_host_bind_port }}:8080 \ From 94830b582bc56226245020b450bb311228c2e299 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 11:58:50 +0100 Subject: [PATCH 053/382] Wording: change collection -> playbook --- docs/configuring-playbook-user-verification-service.md | 6 +++--- group_vars/matrix_servers | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/configuring-playbook-user-verification-service.md b/docs/configuring-playbook-user-verification-service.md index f3d3aa6e..e7fc9570 100644 --- a/docs/configuring-playbook-user-verification-service.md +++ b/docs/configuring-playbook-user-verification-service.md @@ -7,7 +7,7 @@ Currently, the main purpose of this role is to allow Jitsi to authenticate matri **Note**: enabling Matrix User Verification Service, means that the `openid` API endpoints will be exposed on the Matrix Federation port (usually `8448`), even if [federation](configuring-playbook-federation.md) is disabled. -If the Jitsi server is also configured by this collection, all plugging of variables and secrets is handled in `group_vars/matrix_servers`. +If the Jitsi server is also configured by this playbook, all plugging of variables and secrets is handled in `group_vars/matrix_servers`. __Some general concepts of UVS may be helpful to understand the rest, so here they are:__ @@ -18,7 +18,7 @@ UVS can be used to verify two claims: Verifying an OpenID token id done by finding the corresponding Homeserver via '.well-known/matrix/server' for the given domain. The configured `matrix_user_verification_service_uvs_homeserver_url` does **not** factor into this. -By default, this collection only checks against `matrix_server_fqn_matrix`. +By default, this playbook only checks against `matrix_server_fqn_matrix`. Therefore, the request will be made against the public openid API for `matrix_server_fqn_matrix`. Verifying RoomMembership and PowerLevel is done against `matrix_user_verification_service_uvs_homeserver_url` which is by default done via the docker network. @@ -74,7 +74,7 @@ To set your own Token, simply put the following in your host_vars. matrix_user_verification_service_uvs_auth_token: "TOKEN" ``` -In case Jitsi is also managed by this collection and 'matrix' authentication in Jitsi is enabled, this collection will automatically configure Jitsi to use the configured auth token. +In case Jitsi is also managed by this playbook and 'matrix' authentication in Jitsi is enabled, this collection will automatically configure Jitsi to use the configured auth token. ### (Optional) Federation diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 18068bd6..6c785e01 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3207,15 +3207,15 @@ matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matr # URL exposed in the docker network matrix_user_verification_service_container_url: "http://{{ matrix_user_verification_service_container_name }}:3000" -# Set the homeserver URL to the container name if synapse is managed by this collection +# Set the homeserver URL to the container name if synapse is managed by this playbook matrix_user_verification_service_uvs_homeserver_url: "{{ matrix_homeserver_container_url if matrix_synapse_enabled }}" -# If synapse is managed by this collection, we will connect via docker network, which is a private ip. +# If synapse is managed by this playbook, we will connect via docker network, which is a private ip. # Therefore we need to disable IP checks matrix_user_verification_service_uvs_disable_ip_blacklist: "{{'true' if matrix_synapse_enabled else 'false'}}" matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'uvs.auth.token', rounds=655555) | to_uuid }}" -# Pin UVS to only check openId Tokens for the matrix_server_name configured by this collection. +# Pin UVS to only check openId Tokens for the matrix_server_name configured by this playbook. # This is not the homeserverURL, but rather the domain in the matrix "user ID" matrix_user_verification_service_uvs_openid_verify_server_name: "{{ matrix_domain }}" matrix_user_verification_service_uvs_log_level: warning From 8155f780e5291496eaf4f4ab989e676cf33e8a91 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Feb 2023 13:08:11 +0200 Subject: [PATCH 054/382] Add support for reverse-proxying Matric (Client & Federation) via Traefik --- group_vars/matrix_servers | 7 +++++ requirements.yml | 2 +- roles/custom/matrix-base/defaults/main.yml | 3 +++ .../matrix-nginx-proxy/defaults/main.yml | 7 +++++ .../matrix-nginx-proxy/templates/labels.j2 | 26 +++++++++++++++++++ 5 files changed, 44 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index da624d9a..8ce94f52 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2215,6 +2215,7 @@ matrix_nginx_proxy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_config_entrypoint_primary }}" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled: "{{ matrix_client_element_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled }}" @@ -3349,6 +3350,12 @@ devture_traefik_enabled: "{{ matrix_playbook_traefik_role_enabled }}" devture_traefik_uid: "{{ matrix_user_uid }}" devture_traefik_gid: "{{ matrix_user_gid }}" +devture_traefik_additional_entrypoints_auto: + - name: matrix-federation + port: "{{ matrix_federation_public_port }}" + host_bind_port: "{{ matrix_federation_public_port }}" + config: {} + ######################################################################## # # # /com.devture.ansible.role.traefik # diff --git a/requirements.yml b/requirements.yml index 696b2700..e0ff1e81 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,4 +37,4 @@ version: v0.11.1-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: 407af71a3667b1d8083beb10bf22423ecf013f58 + version: b8609fd07c26c89a72fe2934d183af5fd964bc1c diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 63aee58a..025e7363 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -92,6 +92,9 @@ matrix_server_fqn_ntfy: "ntfy.{{ matrix_domain }}" matrix_federation_public_port: 8448 +# The name of the Traefik entrypoint for handling Matrix Federation +matrix_federation_traefik_entrypoint: matrix-federation + # The architecture that your server runs. # Recognized values by us are 'amd64', 'arm32' and 'arm64'. # Not all architectures support all services, so your experience (on non-amd64) may vary. diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 25911a49..e83e9b5b 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -49,6 +49,13 @@ matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_nginx_pro matrix_nginx_proxy_container_labels_traefik_entrypoints: web-secure matrix_nginx_proxy_container_labels_traefik_tls_certResolver: default # noqa var-naming +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_hostname: "{{ matrix_server_fqn_matrix }}" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_hostname }}`)" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" + matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_element_hostname: "{{ matrix_server_fqn_element }}" matrix_nginx_proxy_container_labels_traefik_proxy_element_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 06cc8d1a..407654a7 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -6,6 +6,27 @@ traefik.docker.network={{ matrix_nginx_proxy_container_labels_traefik_docker_net {% endif %} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled %} +# Matrix Client +traefik.http.routers.matrix-nginx-proxy-matrix-client.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule }} +traefik.http.routers.matrix-nginx-proxy-matrix-client.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-matrix-client.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls %} +traefik.http.routers.matrix-nginx-proxy-matrix-client.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-matrix-client.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} + +# Matrix Federation +traefik.http.routers.matrix-nginx-proxy-matrix-federation.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule }} +traefik.http.routers.matrix-nginx-proxy-matrix-federation.service=matrix-nginx-proxy-federation +traefik.http.routers.matrix-nginx-proxy-matrix-federation.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls %} +traefik.http.routers.matrix-nginx-proxy-matrix-federation.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints }} +{% endif %} + + {% if matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled %} # Element traefik.http.routers.matrix-nginx-proxy-element.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_element_rule }} @@ -139,6 +160,11 @@ traefik.http.routers.matrix-nginx-proxy-ntfy.entrypoints={{ matrix_nginx_proxy_c traefik.http.services.matrix-nginx-proxy-web.loadbalancer.server.port=8080 + +{% if matrix_nginx_proxy_proxy_matrix_federation_api_enabled %} +traefik.http.services.matrix-nginx-proxy-federation.loadbalancer.server.port={{ matrix_nginx_proxy_proxy_matrix_federation_port }} +{% endif %} + {% endif %} {{ matrix_nginx_proxy_container_labels_additional_labels }} From f53731756d23079660ee299a0621097dd2936398 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 12:15:54 +0100 Subject: [PATCH 055/382] Change comment Applying the assumption, that synapse is always managed by this playbook. --- group_vars/matrix_servers | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6c785e01..b52c74a3 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3209,8 +3209,7 @@ matrix_user_verification_service_container_url: "http://{{ matrix_user_verifica # Set the homeserver URL to the container name if synapse is managed by this playbook matrix_user_verification_service_uvs_homeserver_url: "{{ matrix_homeserver_container_url if matrix_synapse_enabled }}" -# If synapse is managed by this playbook, we will connect via docker network, which is a private ip. -# Therefore we need to disable IP checks +# We connect via the container network (private IPs), so we need to disable IP checks matrix_user_verification_service_uvs_disable_ip_blacklist: "{{'true' if matrix_synapse_enabled else 'false'}}" matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'uvs.auth.token', rounds=655555) | to_uuid }}" From 6913d368c8a11d61b2de4bb8fbe1adb39c8bb32f Mon Sep 17 00:00:00 2001 From: Jakob S Date: Mon, 6 Feb 2023 12:38:01 +0100 Subject: [PATCH 056/382] Consolidate conditionals into a block, keep image Co-authored-by: Slavi Pantaleev --- .../tasks/setup_uninstall.yml | 42 +++++++------------ 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml index 2a14d96a..5daafd3d 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml @@ -5,31 +5,21 @@ path: "{{ matrix_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" register: matrix_user_verification_service_service_stat -- name: Ensure matrix-user-verification-service is stopped - service: - name: "{{ matrix_user_verification_service_systemd_service_basename }}" - state: stopped - daemon_reload: yes - register: stopping_result - when: "matrix_user_verification_service_service_stat.stat.exists|bool" +- when: matrix_user_verification_service_service_stat.stat.exists | bool + block: + - name: Ensure matrix-user-verification-service is stopped + service: + name: "{{ matrix_user_verification_service_systemd_service_basename }}" + state: stopped + daemon_reload: yes + register: stopping_result -- name: Ensure matrix-user-verification-service.service doesn't exist - file: - path: "{{ matrix_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" - state: absent - when: "matrix_user_verification_service_service_stat.stat.exists|bool" - -- name: Ensure systemd reloaded after matrix-user-verification-service.service removal - service: - daemon_reload: yes - when: "matrix_user_verification_service_service_stat.stat.exists|bool" - -- name: Ensure Matrix user-verification-service paths don't exist - file: - path: "{{ matrix_user_verification_service_base_path }}" - state: absent + - name: Ensure matrix-user-verification-service.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" + state: absent -- name: Ensure user-verification-service Docker image doesn't exist - docker_image: - name: "{{ matrix_user_verification_service_docker_image }}" - state: absent + - name: Ensure Matrix user-verification-service paths don't exist + file: + path: "{{ matrix_user_verification_service_base_path }}" + state: absent From b89f5b7ff5e7e957145c1f2e4a79dba77dc22e89 Mon Sep 17 00:00:00 2001 From: Paul N <92150859+stift-n2@users.noreply.github.com> Date: Mon, 6 Feb 2023 12:47:50 +0100 Subject: [PATCH 057/382] Clarify task name and add user and group to templated env file Co-authored-by: Slavi Pantaleev --- .../matrix-user-verification-service/tasks/setup_install.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml index 5d1d2845..3ce463aa 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml @@ -28,10 +28,12 @@ # TODO # when: "matrix_user_verification_service_container_image_self_build|bool" -- name: write env file +- name: Ensure Matrix User Verification Service env file installed ansible.builtin.template: src: "{{ role_path }}/templates/.env.j2" dest: "{{ matrix_user_verification_service_config_env_file }}" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" mode: 0644 - name: Ensure matrix-user-verification-service.service installed From bf5e633656b8b88bc9dedf7e2be01fcc6f4851b3 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 12:57:20 +0100 Subject: [PATCH 058/382] Remove the self-build stub, because self-build was not implemented --- .../matrix-user-verification-service/defaults/main.yml | 7 +------ .../tasks/setup_install.yml | 6 ------ 2 files changed, 1 insertion(+), 12 deletions(-) diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index a86d6235..2f4b726c 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -15,13 +15,8 @@ matrix_user_verification_service_docker_src_files_path: "{{ matrix_user_verifica matrix_user_verification_service_config_path: "{{ matrix_user_verification_service_base_path }}/config" matrix_user_verification_service_config_env_file: "{{ matrix_user_verification_service_config_path }}/.env" -# Set this to true in order to not use the docker image from docker hub, but rather build locally -matrix_user_verification_service_container_image_self_build: false -matrix_user_verification_service_container_image_self_build_repo: "https://github.com/matrix-org/matrix-user-verification-service.git" -matrix_user_verification_service_container_image_self_build_branch: "{{ matrix_user_verification_service_version }}" - # Docker -matrix_user_verification_service_docker_image_name_prefix: "{{ 'localhost/' if matrix_user_verification_service_container_image_self_build else matrix_container_global_registry_prefix }}" +matrix_user_verification_service_docker_image_name_prefix: "{{ matrix_container_global_registry_prefix }}" matrix_user_verification_service_docker_image: "{{ matrix_user_verification_service_docker_image_name_prefix }}matrixdotorg/matrix-user-verification-service:{{ matrix_user_verification_service_version }}" matrix_user_verification_service_docker_image_force_pull: "{{ matrix_user_verification_service_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml index 3ce463aa..24b8b811 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml @@ -9,7 +9,6 @@ group: "{{ matrix_user_groupname }}" with_items: - { path: "{{ matrix_user_verification_service_config_path }}", when: true } - - { path: "{{ matrix_user_verification_service_docker_src_files_path }}", when: "{{ matrix_user_verification_service_container_image_self_build }}" } when: item.when | bool - name: Ensure Matrix User Verification Service image is pulled @@ -18,16 +17,11 @@ source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" force_source: "{{ matrix_user_verification_service_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_user_verification_service_docker_image_force_pull }}" - when: "not matrix_user_verification_service_container_image_self_build | bool" register: result retries: "{{ devture_playbook_help_container_retries_count }}" delay: "{{ devture_playbook_help_container_retries_delay }}" until: result is not failed -#- block: -# TODO -# when: "matrix_user_verification_service_container_image_self_build|bool" - - name: Ensure Matrix User Verification Service env file installed ansible.builtin.template: src: "{{ role_path }}/templates/.env.j2" From 0e0ae2f3e6c808e392782bc60b168c7a0cb54ba7 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 13:04:06 +0100 Subject: [PATCH 059/382] Assign default log level in role instead of matrix_servers file. --- group_vars/matrix_servers | 1 - roles/custom/matrix-user-verification-service/defaults/main.yml | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index b52c74a3..a5d8c497 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3217,7 +3217,6 @@ matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homese # Pin UVS to only check openId Tokens for the matrix_server_name configured by this playbook. # This is not the homeserverURL, but rather the domain in the matrix "user ID" matrix_user_verification_service_uvs_openid_verify_server_name: "{{ matrix_domain }}" -matrix_user_verification_service_uvs_log_level: warning ###################################################################### # diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index 2f4b726c..fd174394 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -55,7 +55,7 @@ matrix_user_verification_service_uvs_disable_ip_blacklist: false # Log level, defaults to 'info' # See choices here: https://github.com/winstonjs/winston#logging-levels -# matrix_user_verification_service_uvs_log_level: info +matrix_user_verification_service_uvs_log_level: warning ###################################################################### From 07d9ea5e875a217a44ffde6ba6cdb698a6d67ac7 Mon Sep 17 00:00:00 2001 From: Paul N <92150859+stift-n2@users.noreply.github.com> Date: Mon, 6 Feb 2023 13:06:14 +0100 Subject: [PATCH 060/382] Stick to port 3003 instead of changing the port based on the status of grafana. --- docs/configuring-playbook-user-verification-service.md | 2 +- group_vars/matrix_servers | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/configuring-playbook-user-verification-service.md b/docs/configuring-playbook-user-verification-service.md index e7fc9570..d6468b54 100644 --- a/docs/configuring-playbook-user-verification-service.md +++ b/docs/configuring-playbook-user-verification-service.md @@ -45,7 +45,7 @@ For a list of all configuration options see the role defaults [`roles/matrix-use But be aware of all the plugging happening in `group_vars/matrix_servers`. In the default configuration, the UVS Server is only reachable via the docker network, which is fine if e.g. Jitsi is also running in a container on the host. -However, it is possible to expose UVS via setting `matrix_user_verification_service_container_http_host_bind_port`. Be aware that the normally used port (3000) may collide with Grafana. +However, it is possible to expose UVS via setting `matrix_user_verification_service_container_http_host_bind_port`. ### Access token diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index a5d8c497..0a6ae4f3 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3200,9 +3200,9 @@ matrix_user_verification_service_systemd_required_services_list: | # If Jitsi is managed by this playbook we can use the docker network - no need to expose a port. # If Jitsi is not managed by this playbook, or you otherwise have a need for it, you can expose -# matrix-user-verfification-services's client-server port to the local host. -# Note: If grafana is also enabled, the exposed port is changed to 3003. -matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') else '127.0.0.1:' + ('3003' if matrix_nginx_proxy_proxy_grafana_enabled else '3000') }}" +# matrix-user-verfification-services's client-server port to the local host port 3003. +# By default Matrix-User-Verification-Service binds to port 3000, which collides with grafana, therefore this uses port 3003. +matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') else '127.0.0.1:3003' }}" # URL exposed in the docker network matrix_user_verification_service_container_url: "http://{{ matrix_user_verification_service_container_name }}:3000" From 50c1e9d695490d59d3b8b6234b44e69e6c48640c Mon Sep 17 00:00:00 2001 From: Paul N <92150859+stift-n2@users.noreply.github.com> Date: Mon, 6 Feb 2023 13:14:34 +0100 Subject: [PATCH 061/382] Set matrix_user_verification_service_uvs_homeserver_url in the role defaults and updated docs accordingly. --- docs/configuring-playbook-user-verification-service.md | 2 +- group_vars/matrix_servers | 2 -- roles/custom/matrix-user-verification-service/defaults/main.yml | 2 +- 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/configuring-playbook-user-verification-service.md b/docs/configuring-playbook-user-verification-service.md index d6468b54..82e4e84e 100644 --- a/docs/configuring-playbook-user-verification-service.md +++ b/docs/configuring-playbook-user-verification-service.md @@ -49,7 +49,7 @@ However, it is possible to expose UVS via setting `matrix_user_verification_serv ### Access token -The Synapse Access Token is used to verify RoomMembership and PowerLevel against the configured homeserver_url (which is plugged in group_vars). +The Synapse Access Token is used to verify RoomMembership and PowerLevel against `matrix_user_verification_service_uvs_homeserver_url`. We recommend that you create a dedicated Matrix user for uvs (`uvs` is a good username). Follow our [Registering users](registering-users.md) guide to register a user with administration privileges. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 0a6ae4f3..8cd9baa9 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3207,8 +3207,6 @@ matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matr # URL exposed in the docker network matrix_user_verification_service_container_url: "http://{{ matrix_user_verification_service_container_name }}:3000" -# Set the homeserver URL to the container name if synapse is managed by this playbook -matrix_user_verification_service_uvs_homeserver_url: "{{ matrix_homeserver_container_url if matrix_synapse_enabled }}" # We connect via the container network (private IPs), so we need to disable IP checks matrix_user_verification_service_uvs_disable_ip_blacklist: "{{'true' if matrix_synapse_enabled else 'false'}}" diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index fd174394..2b2cbcb2 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -37,7 +37,7 @@ matrix_user_verification_service_systemd_service_name: "{{ matrix_user_verificat # matrix_user_verification_service_uvs_access_token: # homeserver client api url -# matrix_user_verification_service_uvs_homeserver_url: "" +matrix_user_verification_service_uvs_homeserver_url: "{{ matrix_homeserver_container_url }}" # disable check for non private ip range of homeserver. e.g. set to `true` if your homeserver domain resolves to a private ip. matrix_user_verification_service_uvs_disable_ip_blacklist: false From 1d99f17b4a32c724dd7ecf8a1cd36f0b7361f383 Mon Sep 17 00:00:00 2001 From: Paul N <92150859+stift-n2@users.noreply.github.com> Date: Mon, 6 Feb 2023 13:23:11 +0100 Subject: [PATCH 062/382] Disable matrix-user-verification-service in group_vars and update docs accordingly. --- docs/configuring-playbook-jitsi.md | 1 + docs/configuring-playbook-user-verification-service.md | 2 +- group_vars/matrix_servers | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/configuring-playbook-jitsi.md b/docs/configuring-playbook-jitsi.md index f51f8fc1..6c787bdb 100644 --- a/docs/configuring-playbook-jitsi.md +++ b/docs/configuring-playbook-jitsi.md @@ -79,6 +79,7 @@ To enable set this configuration at host level: ```yaml matrix_jitsi_enable_auth: true matrix_jitsi_auth_type: "matrix" +matrix_user_verification_service_enabled: true ``` For more information see also [https://github.com/matrix-org/prosody-mod-auth-matrix-user-verification](https://github.com/matrix-org/prosody-mod-auth-matrix-user-verification). diff --git a/docs/configuring-playbook-user-verification-service.md b/docs/configuring-playbook-user-verification-service.md index 82e4e84e..1990e891 100644 --- a/docs/configuring-playbook-user-verification-service.md +++ b/docs/configuring-playbook-user-verification-service.md @@ -30,7 +30,7 @@ In order to use UVS, an admin token for the configured homeserver must be suppli ## Enable -[Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service) installation is disabled by default unless required by Jitsi (see group_vars/matrix_servers). +[Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service) installation is disabled by default. You can enable it in your configuration file (`inventory/host_vars/matrix./vars.yml`): ```yaml diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 8cd9baa9..1bd1c1d3 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3188,7 +3188,7 @@ matrix_user_creator_users_auto: | matrix_user_verification_service_docker_image: "{{ matrix_user_verification_service_docker_image_name_prefix }}matrixdotorg/matrix-user-verification-service@sha256:d2aabc984dd69d258c91900c36928972d7aaef19d776caa3cd6a0fbc0e307270" # enable if jitsi is managed by this playbook and requires JWT auth -matrix_user_verification_service_enabled: "{{ (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') }}" +matrix_user_verification_service_enabled: false matrix_user_verification_service_systemd_required_services_list: | {{ ['docker.service'] From 6499b6536abe94e28a661b5e1e32751e90d47043 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 15:18:25 +0100 Subject: [PATCH 063/382] Decoupling: Do not use variables user-verification-service role inside the jitsi role. --- group_vars/matrix_servers | 1 + roles/custom/matrix-jitsi/defaults/main.yml | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 1bd1c1d3..cabe3561 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1949,6 +1949,7 @@ matrix_jitsi_etherpad_base: "{{ matrix_etherpad_base_url if matrix_etherpad_enab # Allow verification using JWT and matrix-UVS matrix_jitsi_prosody_auth_matrix_uvs_auth_token: "{{ matrix_user_verification_service_uvs_auth_token }}" +matrix_jitsi_prosody_auth_matrix_uvs_location: "{{ matrix_user_verification_service_container_url }}" matrix_jitsi_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index 1f8502c8..d97689ea 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -56,7 +56,7 @@ matrix_jitsi_prosody_auth_matrix_user_verification_repo_location: "https://githu matrix_jitsi_prosody_auth_matrix_user_verification_repo_target: "{{ matrix_jitsi_prosody_ext_path }}/prosody_auth_matrix_user_verification" matrix_jitsi_prosody_auth_matrix_user_verification_repo_version: "2839499cb03894d8cfc3e5b2219441427cb133d8" # v1.8.0 matrix_jitsi_prosody_auth_matrix_uvs_sync_power_levels: true -matrix_jitsi_prosody_auth_matrix_uvs_location: "{{ matrix_user_verification_service_container_url }}" +matrix_jitsi_prosody_auth_matrix_uvs_location: "" # Should match domain, see https://github.com/vector-im/element-web/pull/15114/commits/0410a6b3be82a41457275e4d1ce879dea146e092 matrix_jitsi_prosody_auth_matrix_jwt_app_id: "{{ matrix_server_fqn_jitsi }}" matrix_jitsi_prosody_auth_matrix_files: From 6b206b3763cebd721fa6cfd5e55c35685e2070e3 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 15:21:10 +0100 Subject: [PATCH 064/382] Move checks into validate_config.yml. --- .../matrix-user-verification-service/tasks/main.yml | 9 ++------- .../tasks/validate_config.yml | 8 ++++++++ 2 files changed, 10 insertions(+), 7 deletions(-) create mode 100644 roles/custom/matrix-user-verification-service/tasks/validate_config.yml diff --git a/roles/custom/matrix-user-verification-service/tasks/main.yml b/roles/custom/matrix-user-verification-service/tasks/main.yml index 0f51d6cc..92686036 100644 --- a/roles/custom/matrix-user-verification-service/tasks/main.yml +++ b/roles/custom/matrix-user-verification-service/tasks/main.yml @@ -1,13 +1,8 @@ --- -- name: verify all necessary variables are present - assert: - that: - - matrix_user_verification_service_uvs_access_token is defined and matrix_user_verification_service_uvs_access_token|length - - matrix_user_verification_service_uvs_homeserver_url is defined and matrix_user_verification_service_uvs_homeserver_url|length - fail_msg: "Missing variable in {{ matrix_user_verification_service_ansible_name }} role" - - block: + - when: matrix_jitsi_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: run_setup | bool and matrix_user_verification_service_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" tags: diff --git a/roles/custom/matrix-user-verification-service/tasks/validate_config.yml b/roles/custom/matrix-user-verification-service/tasks/validate_config.yml new file mode 100644 index 00000000..e4349fa6 --- /dev/null +++ b/roles/custom/matrix-user-verification-service/tasks/validate_config.yml @@ -0,0 +1,8 @@ +--- + +- name: verify all necessary variables are present + assert: + that: + - matrix_user_verification_service_uvs_access_token is defined and matrix_user_verification_service_uvs_access_token|length + - matrix_user_verification_service_uvs_homeserver_url is defined and matrix_user_verification_service_uvs_homeserver_url|length + fail_msg: "Missing variable in {{ matrix_user_verification_service_ansible_name }} role" From d67d8c07f5dde9a8ff702416529bd9d6d5533260 Mon Sep 17 00:00:00 2001 From: Paul N <92150859+stift-n2@users.noreply.github.com> Date: Mon, 6 Feb 2023 15:11:20 +0100 Subject: [PATCH 065/382] Remove remnant comment. --- group_vars/matrix_servers | 1 - 1 file changed, 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index cabe3561..060ffbca 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3188,7 +3188,6 @@ matrix_user_creator_users_auto: | ## FIXME: Needs to be updated when there is a proper release by upstream. matrix_user_verification_service_docker_image: "{{ matrix_user_verification_service_docker_image_name_prefix }}matrixdotorg/matrix-user-verification-service@sha256:d2aabc984dd69d258c91900c36928972d7aaef19d776caa3cd6a0fbc0e307270" -# enable if jitsi is managed by this playbook and requires JWT auth matrix_user_verification_service_enabled: false matrix_user_verification_service_systemd_required_services_list: | {{ From 96dd86d33b943c381d5ffebca256b2bdec9a1780 Mon Sep 17 00:00:00 2001 From: Paul N <92150859+stift-n2@users.noreply.github.com> Date: Mon, 6 Feb 2023 15:19:58 +0100 Subject: [PATCH 066/382] Set default values where sensible and remove unnecessary conditionals in .env.j2. Check for empty string instead of Null to verify if an openid_server_name is pinned. --- ...onfiguring-playbook-user-verification-service.md | 4 ++-- group_vars/matrix_servers | 4 ---- .../defaults/main.yml | 13 +++++++------ .../templates/.env.j2 | 11 +++-------- 4 files changed, 12 insertions(+), 20 deletions(-) diff --git a/docs/configuring-playbook-user-verification-service.md b/docs/configuring-playbook-user-verification-service.md index 1990e891..451f54f4 100644 --- a/docs/configuring-playbook-user-verification-service.md +++ b/docs/configuring-playbook-user-verification-service.md @@ -81,10 +81,10 @@ In case Jitsi is also managed by this playbook and 'matrix' authentication in Ji In theory (however currently untested), UVS can handle federation. Simply set: ```yaml -matrix_user_verification_service_uvs_openid_verify_server_name: ~ +matrix_user_verification_service_uvs_openid_verify_server_name: "" ``` -using host_vars to override the group_vars. +in your host_vars. This will instruct UVS to verify the OpenID token against any domain given in a request. Homeserver discovery is done via '.well-known/matrix/server' of the given domain. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 060ffbca..988af72e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3212,10 +3212,6 @@ matrix_user_verification_service_uvs_disable_ip_blacklist: "{{'true' if matrix_s matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'uvs.auth.token', rounds=655555) | to_uuid }}" -# Pin UVS to only check openId Tokens for the matrix_server_name configured by this playbook. -# This is not the homeserverURL, but rather the domain in the matrix "user ID" -matrix_user_verification_service_uvs_openid_verify_server_name: "{{ matrix_domain }}" - ###################################################################### # # /matrix-user-verification-service diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index 2b2cbcb2..cdef8f39 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -48,14 +48,15 @@ matrix_user_verification_service_uvs_disable_ip_blacklist: false # need have the header "Authorization: Bearer changeme". # matrix_user_verification_service_uvs_auth_token: changeme -# Matrix server name to verify OpenID tokens against. See below section. -# Defaults to empty value which means verification is made against -# whatever Matrix server name passed in with the token -# matrix_user_verification_service_uvs_openid_verify_server_name: matrix.org +# Matrix server name to verify OpenID tokens against. +# Pin UVS to only check openId Tokens for the matrix_server_name configured by this playbook. +# This is not the homeserverURL, but rather the domain in the matrix "user ID" +# UVS can also be instructed to verify against the Matrix server name passed in the token, to enable set to "" +matrix_user_verification_service_uvs_openid_verify_server_name: "{{ matrix_domain }}" -# Log level, defaults to 'info' +# Log level # See choices here: https://github.com/winstonjs/winston#logging-levels -matrix_user_verification_service_uvs_log_level: warning +matrix_user_verification_service_uvs_log_level: info ###################################################################### diff --git a/roles/custom/matrix-user-verification-service/templates/.env.j2 b/roles/custom/matrix-user-verification-service/templates/.env.j2 index b2f2aaab..8119c1e9 100644 --- a/roles/custom/matrix-user-verification-service/templates/.env.j2 +++ b/roles/custom/matrix-user-verification-service/templates/.env.j2 @@ -1,14 +1,9 @@ UVS_ACCESS_TOKEN={{ matrix_user_verification_service_uvs_access_token }} UVS_HOMESERVER_URL={{ matrix_user_verification_service_uvs_homeserver_url }} UVS_DISABLE_IP_BLACKLIST={{ matrix_user_verification_service_uvs_disable_ip_blacklist }} - -{% if matrix_user_verification_service_uvs_auth_token is defined and matrix_user_verification_service_uvs_auth_token|length %} - UVS_AUTH_TOKEN={{ matrix_user_verification_service_uvs_auth_token }} -{% endif %} -{% if matrix_user_verification_service_uvs_openid_verify_server_name is defined and matrix_user_verification_service_uvs_openid_verify_server_name|length %} +UVS_LOG_LEVEL={{ matrix_user_verification_service_uvs_log_level }} +UVS_AUTH_TOKEN={{ matrix_user_verification_service_uvs_auth_token }} +{% if matrix_user_verification_service_uvs_openid_verify_server_name | length > 0 %} UVS_OPENID_VERIFY_SERVER_NAME={{ matrix_user_verification_service_uvs_openid_verify_server_name }} {% endif %} -{% if matrix_user_verification_service_uvs_log_level is defined and matrix_user_verification_service_uvs_log_level|length %} - UVS_LOG_LEVEL={{ matrix_user_verification_service_uvs_log_level }} -{% endif %} From 70bea81df753f281ae844239529b3b6bc94c95cc Mon Sep 17 00:00:00 2001 From: Paul N <92150859+stift-n2@users.noreply.github.com> Date: Mon, 6 Feb 2023 15:59:32 +0100 Subject: [PATCH 067/382] Introduced flags to (1) enable/disable Auth (2) enable/disable openid_server_name pinning. Updated validate_config.yml and added new checks to verify. --- ...ring-playbook-user-verification-service.md | 15 +++++++++--- .../defaults/main.yml | 12 ++++++---- .../tasks/validate_config.yml | 23 ++++++++++++++++--- .../templates/.env.j2 | 6 +++-- 4 files changed, 44 insertions(+), 12 deletions(-) diff --git a/docs/configuring-playbook-user-verification-service.md b/docs/configuring-playbook-user-verification-service.md index 451f54f4..d33c7147 100644 --- a/docs/configuring-playbook-user-verification-service.md +++ b/docs/configuring-playbook-user-verification-service.md @@ -63,9 +63,9 @@ To get an access token for the UVS user, you can follow the documentation on [ho matrix_user_verification_service_uvs_access_token: "YOUR ACCESS TOKEN HERE" ``` -### (Optional) Auth Token +### (Optional) Custom Auth Token -It is possible to set an API Auth Token to restrict access to the UVS. If this is set, anyone making a request to UVS must provide it via the header "Authorization: Bearer TOKEN" +It is possible to set an API Auth Token to restrict access to the UVS. If this is enabled, anyone making a request to UVS must provide it via the header "Authorization: Bearer TOKEN" By default, the token will be derived from `matrix_homeserver_generic_secret_key` in `group_vars/matrix_servers`. To set your own Token, simply put the following in your host_vars. @@ -76,12 +76,21 @@ matrix_user_verification_service_uvs_auth_token: "TOKEN" In case Jitsi is also managed by this playbook and 'matrix' authentication in Jitsi is enabled, this collection will automatically configure Jitsi to use the configured auth token. +### (Optional) Disable Auth +Authorization is enabled by default. To disable set + +```yaml +matrix_user_verification_service_uvs_require_auth: false +``` + +in your host_vars. + ### (Optional) Federation In theory (however currently untested), UVS can handle federation. Simply set: ```yaml -matrix_user_verification_service_uvs_openid_verify_server_name: "" +matrix_user_verification_service_uvs_pin_openid_verify_server_name: false ``` in your host_vars. diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index cdef8f39..6f7be0d1 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -43,13 +43,17 @@ matrix_user_verification_service_uvs_disable_ip_blacklist: false ## OPTIONAL +# Require an Auth-Token with API calls. If set to false, UVS will reply to any API call. +# The Auth-Token is defined via: matrix_user_verification_service_uvs_auth_token +matrix_user_verification_service_uvs_require_auth: true # Auth token to protect the API -# If this is set any calls to the provided API endpoints -# need have the header "Authorization: Bearer changeme". -# matrix_user_verification_service_uvs_auth_token: changeme +# If enabled any calls to the provided API endpoints need have the header "Authorization: Bearer TOKEN". +# A Token will be derived from matrix_homeserver_generic_secret_key in group_vars/matrix_servers +matrix_user_verification_service_uvs_auth_token: '' -# Matrix server name to verify OpenID tokens against. # Pin UVS to only check openId Tokens for the matrix_server_name configured by this playbook. +matrix_user_verification_service_uvs_pin_openid_verify_server_name: true +# Matrix server name to verify OpenID tokens against. # This is not the homeserverURL, but rather the domain in the matrix "user ID" # UVS can also be instructed to verify against the Matrix server name passed in the token, to enable set to "" matrix_user_verification_service_uvs_openid_verify_server_name: "{{ matrix_domain }}" diff --git a/roles/custom/matrix-user-verification-service/tasks/validate_config.yml b/roles/custom/matrix-user-verification-service/tasks/validate_config.yml index e4349fa6..40e9090c 100644 --- a/roles/custom/matrix-user-verification-service/tasks/validate_config.yml +++ b/roles/custom/matrix-user-verification-service/tasks/validate_config.yml @@ -1,8 +1,25 @@ --- -- name: verify all necessary variables are present +- name: Verify homeserver_url is not empty assert: that: - - matrix_user_verification_service_uvs_access_token is defined and matrix_user_verification_service_uvs_access_token|length - - matrix_user_verification_service_uvs_homeserver_url is defined and matrix_user_verification_service_uvs_homeserver_url|length + - matrix_user_verification_service_uvs_homeserver_url|length > 0 fail_msg: "Missing variable in {{ matrix_user_verification_service_ansible_name }} role" + +- name: Verify Auth is configured properly or disabled + assert: + that: + - matrix_user_verification_service_uvs_access_token|length > 0 or not matrix_user_verification_service_uvs_require_auth|bool + fail_msg: "If Auth is enabled, a valid (non empty) TOKEN must be given in 'matrix_user_verification_service_uvs_access_token'." + +- name: Verify server_name for openid verification is given, if pinning a single server_name is enabled. + assert: + that: + - matrix_user_verification_service_uvs_openid_verify_server_name|length > 0 or not matrix_user_verification_service_uvs_pin_openid_verify_server_name|bool + fail_msg: "If pinning a single server_name is enabled, a valid (non empty) server_name must be given in 'matrix_user_verification_service_uvs_openid_verify_server_name'." + +- name: Verify the homeserver implementation is synapse + assert: + that: + - matrix_homeserver_implementation == 'synapse' + fail_msg: "The User-Verification-Service requires Synapse as homeserver implementation" diff --git a/roles/custom/matrix-user-verification-service/templates/.env.j2 b/roles/custom/matrix-user-verification-service/templates/.env.j2 index 8119c1e9..359eed2a 100644 --- a/roles/custom/matrix-user-verification-service/templates/.env.j2 +++ b/roles/custom/matrix-user-verification-service/templates/.env.j2 @@ -2,8 +2,10 @@ UVS_ACCESS_TOKEN={{ matrix_user_verification_service_uvs_access_token }} UVS_HOMESERVER_URL={{ matrix_user_verification_service_uvs_homeserver_url }} UVS_DISABLE_IP_BLACKLIST={{ matrix_user_verification_service_uvs_disable_ip_blacklist }} UVS_LOG_LEVEL={{ matrix_user_verification_service_uvs_log_level }} -UVS_AUTH_TOKEN={{ matrix_user_verification_service_uvs_auth_token }} -{% if matrix_user_verification_service_uvs_openid_verify_server_name | length > 0 %} +{% if matrix_user_verification_service_uvs_require_auth | bool %} + UVS_AUTH_TOKEN={{ matrix_user_verification_service_uvs_auth_token }} +{% endif %} +{% if matrix_user_verification_service_uvs_pin_openid_verify_server_name | bool %} UVS_OPENID_VERIFY_SERVER_NAME={{ matrix_user_verification_service_uvs_openid_verify_server_name }} {% endif %} From 7848d865a50eb95d18f9b330a6e53bbef492986c Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 16:15:06 +0100 Subject: [PATCH 068/382] Also define the vars to be overwritten in group vars within the role vars. --- roles/custom/matrix-jitsi/defaults/main.yml | 2 +- .../custom/matrix-user-verification-service/defaults/main.yml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index d97689ea..25a0e9b2 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -66,7 +66,7 @@ matrix_jitsi_prosody_auth_matrix_files: when: "{{ matrix_jitsi_prosody_auth_matrix_uvs_sync_power_levels }}" # Plugged in group_vars -#matrix_jitsi_prosody_auth_matrix_uvs_auth_token: +matrix_jitsi_prosody_auth_matrix_uvs_auth_token: '' matrix_jitsi_timezone: UTC diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index 6f7be0d1..38304f6d 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -21,8 +21,8 @@ matrix_user_verification_service_docker_image: "{{ matrix_user_verification_serv matrix_user_verification_service_docker_image_force_pull: "{{ matrix_user_verification_service_docker_image.endswith(':latest') }}" matrix_user_verification_service_container_name: "matrix-user-verification-service" -# Normally this would run on port 3000 however that may conflict with grafana. It is thus advised to change this port. -#matrix_user_verification_service_container_http_host_bind_port: +# This will be set in group vars +matrix_user_verification_service_container_http_host_bind_port: '' matrix_user_verification_service_container_extra_arguments: [] # Systemd matrix_user_verification_service_systemd_required_services_list: [] From f3ca4a06322442fe846d514847b1d2b6a1a5e98e Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 16:16:36 +0100 Subject: [PATCH 069/382] Remove unnecessary comment. --- .../defaults/main.yml | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index 38304f6d..c6781fdc 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -61,21 +61,3 @@ matrix_user_verification_service_uvs_openid_verify_server_name: "{{ matrix_domai # Log level # See choices here: https://github.com/winstonjs/winston#logging-levels matrix_user_verification_service_uvs_log_level: info - - -###################################################################### -##### ##### -##### Variables used in this role which are not set by this role ##### -##### ##### -###################################################################### - -# matrix_user_username -# matrix_user_groupname -# matrix_user_uid -# matrix_user_gid -# matrix_container_global_registry_prefix -# matrix_docker_network -# devture_systemd_docker_base_systemd_path -# devture_systemd_docker_base_systemd_unit_home_path -# devture_systemd_docker_base_host_command_sh -# devture_systemd_docker_base_host_command_docker From be634168ac4be065e3bac7868b98fe7a30b75a9e Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 16:29:25 +0100 Subject: [PATCH 070/382] Make the linter happy. --- roles/custom/matrix-jitsi/defaults/main.yml | 2 +- .../tasks/util/setup_jitsi_auth_uvs_install.yml | 4 +++- .../tasks/util/setup_jitsi_auth_uvs_uninstall.yml | 8 +++++--- roles/custom/matrix-jitsi/tasks/validate_config.yml | 1 - .../matrix-user-verification-service/handlers/main.yml | 2 +- .../tasks/setup_install.yml | 2 +- .../tasks/setup_uninstall.yml | 2 +- 7 files changed, 12 insertions(+), 9 deletions(-) diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index 25a0e9b2..66d06e3b 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -54,7 +54,7 @@ matrix_jitsi_ldap_start_tls: false # Auth type: matrix matrix_jitsi_prosody_auth_matrix_user_verification_repo_location: "https://github.com/matrix-org/prosody-mod-auth-matrix-user-verification" matrix_jitsi_prosody_auth_matrix_user_verification_repo_target: "{{ matrix_jitsi_prosody_ext_path }}/prosody_auth_matrix_user_verification" -matrix_jitsi_prosody_auth_matrix_user_verification_repo_version: "2839499cb03894d8cfc3e5b2219441427cb133d8" # v1.8.0 +matrix_jitsi_prosody_auth_matrix_user_verification_repo_version: "2839499cb03894d8cfc3e5b2219441427cb133d8" # v1.8.0 matrix_jitsi_prosody_auth_matrix_uvs_sync_power_levels: true matrix_jitsi_prosody_auth_matrix_uvs_location: "" # Should match domain, see https://github.com/vector-im/element-web/pull/15114/commits/0410a6b3be82a41457275e4d1ce879dea146e092 diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml index 36f33425..f1d9ff21 100644 --- a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml @@ -1,3 +1,5 @@ +--- + - name: Checkout Prosody Auth Matrix User Verification Plugin Repo ansible.builtin.git: repo: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_location }}" @@ -6,7 +8,7 @@ - name: Install Prosody Auth Matrix User Verification Plugin ansible.builtin.copy: - remote_src: yes + remote_src: true src: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_target }}/{{ item.path }}" dest: "{{ matrix_jitsi_prosody_plugins_path }}/{{ item.path }}" with_items: "{{ matrix_jitsi_prosody_auth_matrix_files }}" diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml index 2ba793ec..229ce896 100644 --- a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml @@ -1,3 +1,5 @@ +--- + - name: Remove all files regarding prosody mod auth_matrix_user_verification and .well-known/element/jitsi ansible.builtin.file: path: "{{ item }}" @@ -11,10 +13,10 @@ - name: Remove .well-known/element directory if empty ansible.builtin.command: argv: - - rmdir - - "{{ matrix_static_files_base_path }}/.well-known/element" + - rmdir + - "{{ matrix_static_files_base_path }}/.well-known/element" removes: "{{matrix_static_files_base_path}}/.well-known/element" - ignore_errors: yes + ignore_errors: true - when: matrix_jitsi_prosody_auth_matrix_user_verification_uninstalled.changed block: diff --git a/roles/custom/matrix-jitsi/tasks/validate_config.yml b/roles/custom/matrix-jitsi/tasks/validate_config.yml index 5975a605..f975e4ac 100644 --- a/roles/custom/matrix-jitsi/tasks/validate_config.yml +++ b/roles/custom/matrix-jitsi/tasks/validate_config.yml @@ -45,7 +45,6 @@ or (matrix_jitsi_auth_type == 'matrix' and matrix_jitsi_prosody_auth_matrix_uvs_auth_token|length == 0)) - - name: (Deprecation) Catch and report renamed settings ansible.builtin.fail: msg: >- diff --git a/roles/custom/matrix-user-verification-service/handlers/main.yml b/roles/custom/matrix-user-verification-service/handlers/main.yml index a1ad1f31..b9ee66d7 100644 --- a/roles/custom/matrix-user-verification-service/handlers/main.yml +++ b/roles/custom/matrix-user-verification-service/handlers/main.yml @@ -2,5 +2,5 @@ - name: Ensure systemd reloaded after matrix-user-verification-service.service installation service: - daemon_reload: yes + daemon_reload: true listen: "reload matrix-user-verification-service" diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml index 24b8b811..d095c410 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml @@ -8,7 +8,7 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" with_items: - - { path: "{{ matrix_user_verification_service_config_path }}", when: true } + - {path: "{{ matrix_user_verification_service_config_path }}", when: true} when: item.when | bool - name: Ensure Matrix User Verification Service image is pulled diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml index 5daafd3d..bc09f7ef 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml @@ -11,7 +11,7 @@ service: name: "{{ matrix_user_verification_service_systemd_service_basename }}" state: stopped - daemon_reload: yes + daemon_reload: true register: stopping_result - name: Ensure matrix-user-verification-service.service doesn't exist From 6cffec14eabf6cf1001684ff7bd88c3c67ba6444 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 16:36:49 +0100 Subject: [PATCH 071/382] fixup! Remove the self-build stub, because self-build was not implemented --- roles/custom/matrix-user-verification-service/defaults/main.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index c6781fdc..52b30a28 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -10,8 +10,6 @@ matrix_user_verification_service_version: "v2.0.0" # Paths matrix_user_verification_service_base_path: "{{ matrix_base_data_path }}/user-verification-service" -# We need the docker src directory to be named user_verification_service. See: https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/588 -matrix_user_verification_service_docker_src_files_path: "{{ matrix_user_verification_service_base_path }}/docker-src/user-verification-service" matrix_user_verification_service_config_path: "{{ matrix_user_verification_service_base_path }}/config" matrix_user_verification_service_config_env_file: "{{ matrix_user_verification_service_config_path }}/.env" From 6a205a83f616f0217b2cbd90f1c193b5ef19bf64 Mon Sep 17 00:00:00 2001 From: jakicoll Date: Mon, 6 Feb 2023 17:20:13 +0100 Subject: [PATCH 072/382] Change renamed variables matrix_systemd_path -> devture_systemd_docker_base_systemd_path --- .../tasks/setup_uninstall.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml index bc09f7ef..172bf186 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml @@ -2,7 +2,7 @@ - name: Check existence of matrix-user-verification-service service stat: - path: "{{ matrix_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" + path: "{{ devture_systemd_docker_base_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" register: matrix_user_verification_service_service_stat - when: matrix_user_verification_service_service_stat.stat.exists | bool @@ -16,7 +16,7 @@ - name: Ensure matrix-user-verification-service.service doesn't exist file: - path: "{{ matrix_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" + path: "{{ devture_systemd_docker_base_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" state: absent - name: Ensure Matrix user-verification-service paths don't exist From 66baef5bf6c1dc89580204f5e5368b346e0c80c3 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 7 Feb 2023 08:48:50 +0200 Subject: [PATCH 073/382] Fix matrix-synapse-reverse-proxy-companion.service stopping during uninstallation Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2444 --- .../tasks/setup_uninstall.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml b/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml index bb1b534c..7b820b35 100644 --- a/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml @@ -9,7 +9,7 @@ block: - name: Ensure matrix-synapse-reverse-proxy-companion.service is stopped ansible.builtin.service: - name: matrix_synapse_reverse_proxy_companion_service_stat + name: matrix-synapse-reverse-proxy-companion state: stopped enabled: false daemon_reload: true From 6c17671abd3e152300609a6abf5851d421637aa1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 7 Feb 2023 10:45:19 +0200 Subject: [PATCH 074/382] Upgrade synapse-admin (0.8.6 -> 0.8.7) and drop reverse-proxy workaround Related to 6a31fba346d9da434, 6a31fba346d9d. Related to https://github.com/Awesome-Technologies/synapse-admin/issues/322 --- roles/custom/matrix-synapse-admin/defaults/main.yml | 2 +- .../tasks/inject_into_nginx_proxy.yml | 9 --------- 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index ae77a570..0f5c7af6 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -9,7 +9,7 @@ matrix_synapse_admin_container_image_self_build_repo: "https://github.com/Awesom matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src" -matrix_synapse_admin_version: 0.8.6 +matrix_synapse_admin_version: 0.8.7 matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:{{ matrix_synapse_admin_version }}" matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml index 2c25ff47..6a4af859 100644 --- a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml @@ -24,15 +24,6 @@ {# Generic configuration for use outside of our container setup #} proxy_pass http://127.0.0.1:8766/$1; {% endif %} - - {# - Workaround synapse-admin serving all assets at /static. - See: https://github.com/Awesome-Technologies/synapse-admin/issues/322 - #} - sub_filter_once off; - sub_filter "/static/" "{{ matrix_synapse_admin_public_endpoint }}/static/"; - sub_filter "/favicon.ico" "{{ matrix_synapse_admin_public_endpoint }}/favicon.ico"; - sub_filter "/manifest.json" "{{ matrix_synapse_admin_public_endpoint }}/manifest.json"; } - name: Register Synapse Admin proxying configuration with matrix-nginx-proxy From 2b9061a5d361b99f344e9198df5141367c909113 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 7 Feb 2023 11:02:02 +0200 Subject: [PATCH 075/382] Add support for reverse-proxying the base domain via Traefik --- roles/custom/matrix-nginx-proxy/defaults/main.yml | 5 +++++ roles/custom/matrix-nginx-proxy/templates/labels.j2 | 12 ++++++++++++ 2 files changed, 17 insertions(+) diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index e83e9b5b..b60b6194 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -49,6 +49,11 @@ matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_nginx_pro matrix_nginx_proxy_container_labels_traefik_entrypoints: web-secure matrix_nginx_proxy_container_labels_traefik_tls_certResolver: default # noqa var-naming +matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_enabled: "{{ matrix_nginx_proxy_base_domain_serving_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_hostname: "{{ matrix_nginx_proxy_base_domain_hostname }}" +matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_hostname }}`)" + matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_matrix_hostname: "{{ matrix_server_fqn_matrix }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 407654a7..b6994617 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -6,6 +6,18 @@ traefik.docker.network={{ matrix_nginx_proxy_container_labels_traefik_docker_net {% endif %} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_enabled %} +# Base domain +traefik.http.routers.matrix-nginx-proxy-base-domain.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_rule }} +traefik.http.routers.matrix-nginx-proxy-base-domain.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-base-domain.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_tls %} +traefik.http.routers.matrix-nginx-proxy-base-domain.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-base-domain.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + + {% if matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled %} # Matrix Client traefik.http.routers.matrix-nginx-proxy-matrix-client.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule }} From 2eb2ad0ad72780883e0c57ade07bbe5ea9d44f05 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 7 Feb 2023 09:06:00 +0000 Subject: [PATCH 076/382] Update heisenbridge 1.14.1 -> 1.14.2 --- roles/custom/matrix-bridge-heisenbridge/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml b/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml index d51bb913..ba5471cc 100644 --- a/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml +++ b/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml @@ -4,7 +4,7 @@ matrix_heisenbridge_enabled: true -matrix_heisenbridge_version: 1.14.1 +matrix_heisenbridge_version: 1.14.2 matrix_heisenbridge_docker_image: "{{ matrix_container_global_registry_prefix }}hif1/heisenbridge:{{ matrix_heisenbridge_version }}" matrix_heisenbridge_docker_image_force_pull: "{{ matrix_heisenbridge_docker_image.endswith(':latest') }}" From c07630ed51f05117659635f45405d946d97d00d6 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 8 Feb 2023 16:05:38 +0200 Subject: [PATCH 077/382] Add com.devture.ansible.role.traefik_certs_dumper role With this, other roles (like Coturn, Postmoogle) will be able to use SSL certificates extracted from Traefik via https://github.com/ldez/traefik-certs-dumper --- group_vars/matrix_servers | 30 ++++++++++++++++++++++++++++++ playbooks/matrix.yml | 3 +++ requirements.yml | 5 ++++- 3 files changed, 37 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 8ce94f52..08e3bf4e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -38,6 +38,8 @@ matrix_playbook_traefik_role_enabled: false # installed in another way. matrix_playbook_traefik_labels_enabled: false +matrix_playbook_traefik_certs_dumper_role_enabled: "{{ matrix_playbook_traefik_role_enabled }}" + # Controls the additional network that reverse-proxyable services will be connected to. matrix_playbook_reverse_proxyable_services_additional_network: "{{ devture_traefik_container_network if devture_traefik_enabled else '' }}" @@ -344,6 +346,8 @@ devture_systemd_service_manager_services_list_auto: | ([{'name': 'matrix-synapse-reverse-proxy-companion.service', 'priority': 1500, 'groups': ['matrix', 'homeservers', 'synapse', 'reverse-proxies']}] if matrix_synapse_reverse_proxy_companion_enabled else []) + ([{'name': 'devture-traefik.service', 'priority': 3000, 'groups': ['matrix', 'traefik', 'reverse-proxies']}] if matrix_playbook_traefik_role_enabled else []) + + + ([{'name': (devture_traefik_certs_dumper_identifier + '.service'), 'priority': 3500, 'groups': ['matrix', 'traefik-certs-dumper']}] if matrix_playbook_traefik_certs_dumper_role_enabled else []) }} ######################################################################## @@ -3361,3 +3365,29 @@ devture_traefik_additional_entrypoints_auto: # /com.devture.ansible.role.traefik # # # ######################################################################## + + +######################################################################## +# # +# com.devture.ansible.role.traefik_certs_dumper # +# # +######################################################################## + +# To completely disable the Traefik certs dumper role from running, use `matrix_playbook_traefik_certs_dumper_role_enabled: false`. +# See the comment there for more details about why we have both `devture_traefik_certs_dumper_enabled` and `matrix_playbook_traefik_certs_dumper_role_enabled`. +devture_traefik_certs_dumper_enabled: "{{ matrix_playbook_traefik_certs_dumper_role_enabled and devture_traefik_enabled }}" + +devture_traefik_certs_dumper_identifier: matrix-traefik-certs-dumper + +devture_traefik_certs_dumper_base_path: "{{ matrix_base_data_path }}/traefik-certs-dumper" + +devture_traefik_certs_dumper_uid: "{{ matrix_user_uid }}" +devture_traefik_certs_dumper_gid: "{{ matrix_user_gid }}" + +devture_traefik_certs_dumper_ssl_dir_path: "{{ devture_traefik_ssl_dir_path }}" + +######################################################################## +# # +# /com.devture.ansible.role.traefik_certs_dumper # +# # +######################################################################## diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 392a3a0e..3199f8cf 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -118,6 +118,9 @@ - when: matrix_playbook_traefik_role_enabled | bool role: galaxy/com.devture.ansible.role.traefik + - when: matrix_playbook_traefik_certs_dumper_role_enabled | bool + role: galaxy/com.devture.ansible.role.traefik_certs_dumper + - when: devture_systemd_service_manager_enabled | bool role: galaxy/com.devture.ansible.role.systemd_service_manager diff --git a/requirements.yml b/requirements.yml index e0ff1e81..b0dd8d1a 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,4 +37,7 @@ version: v0.11.1-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: b8609fd07c26c89a72fe2934d183af5fd964bc1c + version: c90a0adcdc1de3d00d256e794b095ffbf466bb05 + +- src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git + version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 From d44d4b637f1d01b1957c451e1bf90339e9f7f3bb Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 8 Feb 2023 16:06:46 +0200 Subject: [PATCH 078/382] Allow Coturn to work with SSL certificates extracted from Traefik --- group_vars/matrix_servers | 90 ++++++++++++++----- .../systemd/matrix-coturn.service.j2 | 2 +- 2 files changed, 70 insertions(+), 22 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 08e3bf4e..1d75d2c0 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -43,6 +43,10 @@ matrix_playbook_traefik_certs_dumper_role_enabled: "{{ matrix_playbook_traefik_r # Controls the additional network that reverse-proxyable services will be connected to. matrix_playbook_reverse_proxyable_services_additional_network: "{{ devture_traefik_container_network if devture_traefik_enabled else '' }}" +matrix_playbook_reverse_proxy_type: "{{ 'traefik' if matrix_playbook_traefik_role_enabled else ('nginx-proxy' if matrix_nginx_proxy_enabled else 'none') }}" + +matrix_playbook_ssl_retrieval_method: "{{ 'lets-encrypt' if matrix_playbook_traefik_certs_dumper_role_enabled else matrix_ssl_retrieval_method }}" + ######################################################################## # # # /Playbook # @@ -1908,18 +1912,62 @@ matrix_coturn_turn_external_ip_address: "{{ ansible_host }}" matrix_coturn_turn_static_auth_secret: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'coturn.sas', rounds=655555) | to_uuid }}" -matrix_coturn_tls_enabled: "{{ matrix_ssl_retrieval_method != 'none' }}" -matrix_coturn_tls_cert_path: "{{ matrix_ssl_config_dir_path }}/live/{{ matrix_server_fqn_matrix }}/fullchain.pem" -matrix_coturn_tls_key_path: "{{ matrix_ssl_config_dir_path }}/live/{{ matrix_server_fqn_matrix }}/privkey.pem" +matrix_coturn_tls_enabled: "{{ matrix_playbook_ssl_retrieval_method != 'none' }}" + +matrix_coturn_tls_cert_path: |- + {{ + { + 'nginx-proxy': '/fullchain.pem', + 'traefik': '/certificate.crt', + }[matrix_playbook_reverse_proxy_type] + }} + +matrix_coturn_tls_key_path: |- + {{ + { + 'nginx-proxy': '/privkey.pem', + 'traefik': '/privatekey.key', + }[matrix_playbook_reverse_proxy_type] + }} + matrix_coturn_container_additional_volumes: | {{ - ([] if matrix_ssl_retrieval_method == 'none' else [ - { - 'src': matrix_ssl_config_dir_path, - 'dst': matrix_ssl_config_dir_path, - 'options': 'ro', - } - ]) + ( + [ + { + 'src': (matrix_ssl_config_dir_path + '/live/' + matrix_server_fqn_matrix + '/fullchain.pem'), + 'dst': '/fullchain.pem', + 'options': 'ro', + }, + { + 'src': (matrix_ssl_config_dir_path + '/live/' + matrix_server_fqn_matrix + '/privkey.pem'), + 'dst': '/privkey.pem', + 'options': 'ro', + }, + ] if matrix_playbook_reverse_proxy_type == 'nginx-proxy' else [] + ) + + + ( + [ + { + 'src': (devture_traefik_certs_dumper_dumped_certificates_dir_path + '/' + matrix_server_fqn_matrix + '/certificate.crt'), + 'dst': '/certificate.crt', + 'options': 'ro', + }, + { + 'src': (devture_traefik_certs_dumper_dumped_certificates_dir_path + '/' + matrix_server_fqn_matrix + '/privatekey.key'), + 'dst': '/privatekey.key', + 'options': 'ro', + }, + ] if matrix_playbook_reverse_proxy_type == 'traefik' and matrix_playbook_traefik_certs_dumper_role_enabled else [] + ) + }} + +matrix_coturn_systemd_required_services_list: | + {{ + ['docker.service'] + + + ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type == 'traefik' and matrix_playbook_traefik_certs_dumper_role_enabled else []) }} ###################################################################### @@ -2146,7 +2194,7 @@ matrix_ma1sd_threepid_medium_email_connectors_smtp_host: "matrix-mailer" matrix_ma1sd_threepid_medium_email_connectors_smtp_port: 8025 matrix_ma1sd_threepid_medium_email_connectors_smtp_tls: 0 -matrix_ma1sd_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_ma1sd_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" matrix_ma1sd_systemd_required_services_list: | {{ @@ -2281,12 +2329,12 @@ matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled: "{{ matrix_ma1sd_ matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container }}" matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container }}" -matrix_nginx_proxy_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_nginx_proxy_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" # OCSP stapling does not make sense when self-signed certificates are used. # See https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1073 # and https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1074 -matrix_nginx_proxy_ocsp_stapling_enabled: "{{ matrix_ssl_retrieval_method != 'self-signed' }}" +matrix_nginx_proxy_ocsp_stapling_enabled: "{{ matrix_playbook_ssl_retrieval_method != 'self-signed' }}" matrix_nginx_proxy_systemd_wanted_services_list: | {{ @@ -2768,7 +2816,7 @@ matrix_client_element_integrations_rest_url: "{{ matrix_dimension_integrations_r matrix_client_element_integrations_widgets_urls: "{{ matrix_dimension_integrations_widgets_urls if matrix_dimension_enabled else ['https://scalar.vector.im/api'] }}" matrix_client_element_integrations_jitsi_widget_url: "{{ matrix_dimension_integrations_jitsi_widget_url if matrix_dimension_enabled else 'https://scalar.vector.im/api/widgets/jitsi.html' }}" -matrix_client_element_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_client_element_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" matrix_client_element_registration_enabled: "{{ matrix_synapse_enable_registration }}" @@ -2806,7 +2854,7 @@ matrix_client_hydrogen_container_http_host_bind_port: "{{ '' if matrix_nginx_pro matrix_client_hydrogen_default_hs_url: "{{ matrix_homeserver_url }}" -matrix_client_hydrogen_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_client_hydrogen_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" ###################################################################### # @@ -2831,7 +2879,7 @@ matrix_client_cinny_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_ matrix_client_cinny_default_hs_url: "{{ matrix_homeserver_url }}" -matrix_client_cinny_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_client_cinny_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" ###################################################################### # @@ -2899,7 +2947,7 @@ matrix_synapse_turn_uris: | [ 'turns:' + matrix_server_fqn_matrix + '?transport=udp', 'turns:' + matrix_server_fqn_matrix + '?transport=tcp', - ] if matrix_coturn_enabled and matrix_coturn_tls_enabled and matrix_ssl_retrieval_method != 'lets-encrypt' else [] + ] if matrix_coturn_enabled and matrix_coturn_tls_enabled and matrix_playbook_ssl_retrieval_method != 'lets-encrypt' else [] + [ 'turn:' + matrix_server_fqn_matrix + '?transport=udp', @@ -2909,7 +2957,7 @@ matrix_synapse_turn_uris: | matrix_synapse_turn_shared_secret: "{{ matrix_coturn_turn_static_auth_secret if matrix_coturn_enabled else '' }}" -matrix_synapse_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_synapse_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" matrix_synapse_systemd_required_services_list: | {{ @@ -3186,7 +3234,7 @@ matrix_registration_shared_secret: |- matrix_registration_server_location: "{{ matrix_homeserver_container_url }}" -matrix_registration_api_validate_certs: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_registration_api_validate_certs: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" matrix_registration_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" @@ -3251,9 +3299,9 @@ matrix_dendrite_client_api_turn_uris: | matrix_dendrite_client_api_turn_shared_secret: "{{ matrix_coturn_turn_static_auth_secret if matrix_coturn_enabled else '' }}" -matrix_dendrite_disable_tls_validation: "{{ true if matrix_ssl_retrieval_method == 'self-signed' else false }}" +matrix_dendrite_disable_tls_validation: "{{ true if matrix_playbook_ssl_retrieval_method == 'self-signed' else false }}" -matrix_dendrite_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_dendrite_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" matrix_dendrite_trusted_id_servers: "{{ [matrix_server_fqn_matrix] if matrix_ma1sd_enabled else ['matrix.org', 'vector.im'] }}" diff --git a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 index 8c0272cf..523ad1cc 100644 --- a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 +++ b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 @@ -35,7 +35,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endif %} --mount type=bind,src={{ matrix_coturn_config_path }},dst=/turnserver.conf,ro \ {% for volume in matrix_coturn_container_additional_volumes %} - -v {{ volume.src }}:{{ volume.dst }}:{{ volume.options }} \ + --mount type=bind,src={{ volume.src }},dst={{ volume.dst }}{{ (',' + volume.options) if volume.options else '' }} \ {% endfor %} {% for arg in matrix_coturn_container_extra_arguments %} {{ arg }} \ From ddf6b2d4eead44547bdc804d10b1940fbd3a6890 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 8 Feb 2023 16:12:19 +0200 Subject: [PATCH 079/382] Handle matrix_playbook_reverse_proxy_type being "none" when deciding on Coturn certificate parameters --- group_vars/matrix_servers | 2 ++ 1 file changed, 2 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 1d75d2c0..92d729d7 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1919,6 +1919,7 @@ matrix_coturn_tls_cert_path: |- { 'nginx-proxy': '/fullchain.pem', 'traefik': '/certificate.crt', + 'none': '', }[matrix_playbook_reverse_proxy_type] }} @@ -1927,6 +1928,7 @@ matrix_coturn_tls_key_path: |- { 'nginx-proxy': '/privkey.pem', 'traefik': '/privatekey.key', + 'none': '', }[matrix_playbook_reverse_proxy_type] }} From 9a71a5696ba58f3c9278e52a7d926aa8edf58796 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 8 Feb 2023 16:45:03 +0200 Subject: [PATCH 080/382] Allow Postmoogle to work with SSL certificates extracted from Traefik --- group_vars/matrix_servers | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 92d729d7..d146b252 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1693,9 +1693,40 @@ matrix_bot_postmoogle_enabled: false matrix_bot_postmoogle_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm64'] }}" -matrix_bot_postmoogle_ssl_path: "{{ matrix_ssl_config_dir_path }}" -matrix_bot_postmoogle_tls_cert: "{% for domain in matrix_bot_postmoogle_domains %}/ssl/live/{{ domain }}/fullchain.pem {% endfor %}" -matrix_bot_postmoogle_tls_key: "{% for domain in matrix_bot_postmoogle_domains %}/ssl/live/{{ domain }}/privkey.pem {% endfor %}" +matrix_bot_postmoogle_ssl_path: |- + {{ + { + 'nginx-proxy': (matrix_ssl_config_dir_path if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'traefik': devture_traefik_certs_dumper_dumped_certificates_dir_path, + 'none': '', + }[matrix_playbook_reverse_proxy_type] + }} + +matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert: "{% for domain in matrix_bot_postmoogle_domains %}/ssl/live/{{ domain }}/fullchain.pem {% endfor %}" +matrix_playbook_bot_postmoogle_nginx_proxy_key: "{% for domain in matrix_bot_postmoogle_domains %}/ssl/live/{{ domain }}/privkey.pem {% endfor %}" + +matrix_playbook_bot_postmoogle_traefik_tls_cert: "{% for domain in matrix_bot_postmoogle_domains %}/ssl/{{ domain }}/certificate.crt {% endfor %}" +matrix_playbook_bot_postmoogle_traefik_key: "{% for domain in matrix_bot_postmoogle_domains %}/ssl/{{ domain }}/privatekey.key {% endfor %}" + +matrix_bot_postmoogle_tls_cert: |- + {{ + { + 'nginx-proxy': (matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'traefik': matrix_playbook_bot_postmoogle_traefik_tls_cert, + 'none': '', + }[matrix_playbook_reverse_proxy_type] + }} + +matrix_bot_postmoogle_tls_key: |- + {{ + { + 'nginx-proxy': (matrix_playbook_bot_postmoogle_nginx_proxy_key if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'traefik': matrix_playbook_bot_postmoogle_traefik_key, + 'none': '', + }[matrix_playbook_reverse_proxy_type] + }} + +matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services: "{% for domain in matrix_bot_postmoogle_domains %}{{ devture_traefik_certs_dumper_identifier }}-wait-for-domain@{{ domain }}.service {% endfor %}" matrix_bot_postmoogle_systemd_required_services_list: | {{ @@ -1704,6 +1735,8 @@ matrix_bot_postmoogle_systemd_required_services_list: | ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services | trim | split(' ') if matrix_playbook_reverse_proxy_type == 'traefik' and matrix_playbook_traefik_certs_dumper_role_enabled else []) }} # Postgres is the default, except if not using internal Postgres server From 49a1985750aa38786935ec4a70d504199e7831ce Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 8 Feb 2023 16:45:52 +0200 Subject: [PATCH 081/382] Fix Postmoogle systemd service description --- .../templates/systemd/matrix-bot-postmoogle.service.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 b/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 index f2610600..ab1177f6 100644 --- a/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 +++ b/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 @@ -1,6 +1,6 @@ #jinja2: lstrip_blocks: "True" [Unit] -Description=Matrix helpdesk bot +Description=Matrix Postmoogle bot {% for service in matrix_bot_postmoogle_systemd_required_services_list %} Requires={{ service }} After={{ service }} From 563cf1a4bafcfe84d3166133e6ed41140949d050 Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Wed, 8 Feb 2023 16:44:12 +0100 Subject: [PATCH 082/382] Initial commit for draupnir. main.yml is not included due to that its changed separately. --- .../custom/matrix-bot-draupnir/tasks/main.yml | 20 ++ .../tasks/setup_install.yml | 74 ++++++ .../tasks/setup_uninstall.yml | 25 ++ .../tasks/validate_config.yml | 9 + .../templates/production.yaml.j2 | 246 ++++++++++++++++++ .../systemd/matrix-bot-mjolnir.service.j2 | 42 +++ 6 files changed, 416 insertions(+) create mode 100644 roles/custom/matrix-bot-draupnir/tasks/main.yml create mode 100644 roles/custom/matrix-bot-draupnir/tasks/setup_install.yml create mode 100644 roles/custom/matrix-bot-draupnir/tasks/setup_uninstall.yml create mode 100644 roles/custom/matrix-bot-draupnir/tasks/validate_config.yml create mode 100644 roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 create mode 100644 roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-mjolnir.service.j2 diff --git a/roles/custom/matrix-bot-draupnir/tasks/main.yml b/roles/custom/matrix-bot-draupnir/tasks/main.yml new file mode 100644 index 00000000..686fe298 --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/main.yml @@ -0,0 +1,20 @@ +--- + +- block: + - when: matrix_bot_draupnir_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_bot_draupnir_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + tags: + - setup-all + - setup-bot-draupnir + - install-all + - install-bot-draupnir + +- block: + - when: not matrix_bot_draupnir_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + tags: + - setup-all + - setup-bot-draupnir diff --git a/roles/custom/matrix-bot-draupnir/tasks/setup_install.yml b/roles/custom/matrix-bot-draupnir/tasks/setup_install.yml new file mode 100644 index 00000000..4808f71f --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/setup_install.yml @@ -0,0 +1,74 @@ +--- + +- ansible.builtin.set_fact: + matrix_bot_draupnir_requires_restart: false + +- name: Ensure matrix-bot-draupnir paths exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - {path: "{{ matrix_bot_draupnir_base_path }}", when: true} + - {path: "{{ matrix_bot_draupnir_config_path }}", when: true} + - {path: "{{ matrix_bot_draupnir_data_path }}", when: true} + - {path: "{{ matrix_bot_draupnir_docker_src_files_path }}", when: "{{ matrix_bot_draupnir_container_image_self_build }}"} + when: "item.when | bool" + +- name: Ensure draupnir Docker image is pulled + community.docker.docker_image: + name: "{{ matrix_bot_draupnir_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_bot_draupnir_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_draupnir_docker_image_force_pull }}" + when: "not matrix_bot_draupnir_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure draupnir repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_bot_draupnir_container_image_self_build_repo }}" + dest: "{{ matrix_bot_draupnir_docker_src_files_path }}" + version: "{{ matrix_bot_draupnir_docker_image.split(':')[1] }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_bot_draupnir_git_pull_results + when: "matrix_bot_draupnir_container_image_self_build | bool" + +- name: Ensure draupnir Docker image is built + community.docker.docker_image: + name: "{{ matrix_bot_draupnir_docker_image }}" + source: build + force_source: "{{ matrix_bot_draupnir_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_bot_draupnir_docker_src_files_path }}" + pull: true + when: "matrix_bot_draupnir_container_image_self_build | bool" + +- name: Ensure matrix-bot-draupnir config installed + ansible.builtin.copy: + content: "{{ matrix_bot_draupnir_configuration | to_nice_yaml(indent=2, width=999999) }}" + dest: "{{ matrix_bot_draupnir_config_path }}/production.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-bot-draupnir.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-bot-draupnir.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-draupnir.service" + mode: 0644 + register: matrix_bot_draupnir_systemd_service_result + +- name: Ensure matrix-bot-draupnir.service restarted, if necessary + ansible.builtin.service: + name: "matrix-bot-draupnir.service" + state: restarted + daemon_reload: true + when: "matrix_bot_draupnir_requires_restart | bool" diff --git a/roles/custom/matrix-bot-draupnir/tasks/setup_uninstall.yml b/roles/custom/matrix-bot-draupnir/tasks/setup_uninstall.yml new file mode 100644 index 00000000..10583a0b --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/setup_uninstall.yml @@ -0,0 +1,25 @@ +--- + +- name: Check existence of matrix-bot-draupnir service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-draupnir.service" + register: matrix_bot_draupnir_service_stat + +- when: matrix_bot_draupnir_service_stat.stat.exists | bool + block: + - name: Ensure matrix-bot-draupnir is stopped + ansible.builtin.service: + name: matrix-bot-draupnir + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-bot-draupnir.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-draupnir.service" + state: absent + + - name: Ensure matrix-bot-draupnir paths don't exist + ansible.builtin.file: + path: "{{ matrix_bot_draupnir_base_path }}" + state: absent diff --git a/roles/custom/matrix-bot-draupnir/tasks/validate_config.yml b/roles/custom/matrix-bot-draupnir/tasks/validate_config.yml new file mode 100644 index 00000000..b3828189 --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/validate_config.yml @@ -0,0 +1,9 @@ +--- + +- name: Fail if required variables are undefined + ansible.builtin.fail: + msg: "The `{{ item }}` variable must be defined and have a non-null value." + with_items: + - "matrix_bot_draupnir_access_token" + - "matrix_bot_draupnir_management_room" + when: "vars[item] == '' or vars[item] is none" diff --git a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 new file mode 100644 index 00000000..06d88f48 --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 @@ -0,0 +1,246 @@ +# Endpoint URL that draupnir uses to interact with the matrix homeserver (client-server API), +# set this to the pantalaimon URL if you're using that. +homeserverUrl: "{{ matrix_homeserver_url }}" + +# Endpoint URL that draupnir could use to fetch events related to reports (client-server API and /_synapse/), +# only set this to the public-internet homeserver client API URL, do NOT set this to the pantalaimon URL. +rawHomeserverUrl: "{{ matrix_homeserver_url }}" + +# Matrix Access Token to use, draupnir will only use this if pantalaimon.use is false. +accessToken: "{{ matrix_bot_draupnir_access_token }}" + +# Options related to Pantalaimon (https://github.com/matrix-org/pantalaimon) +#pantalaimon: +# # Whether or not draupnir will use pantalaimon to access the matrix homeserver, +# # set to `true` if you're using pantalaimon. +# # +# # Be sure to point homeserverUrl to the pantalaimon instance. +# # +# # draupnir will log in using the given username and password once, +# # then store the resulting access token in a file under dataPath. +# use: false +# +# # The username to login with. +# username: draupnir +# +# # The password draupnir will login with. +# # +# # After successfully logging in once, this will be ignored, so this value can be blanked after first startup. +# password: your_password + +# The path draupnir will store its state/data in, leave default ("/data/storage") when using containers. +dataPath: "/data" + +# If true (the default), draupnir will only accept invites from users present in managementRoom. +autojoinOnlyIfManager: true + +# If `autojoinOnlyIfManager` is false, only the members in this space can invite +# the bot to new rooms. +#acceptInvitesFromSpace: "!example:example.org" + +# Whether draupnir should report ignored invites to the management room (if autojoinOnlyIfManager is true). +recordIgnoredInvites: false + +# The room ID (or room alias) of the management room, anyone in this room can issue commands to draupnir. +# +# draupnir has no more granular access controls other than this, be sure you trust everyone in this room - secure it! +# +# This should be a room alias or room ID - not a matrix.to URL. +# +# Note: By default, draupnir is fairly verbose - expect a lot of messages in this room. +# (see verboseLogging to adjust this a bit.) +managementRoom: "{{ matrix_bot_draupnir_management_room }}" + +# Whether draupnir should log a lot more messages in the room, +# mainly involves "all-OK" messages, and debugging messages for when draupnir checks bans in a room. +verboseLogging: false + +# The log level of terminal (or container) output, +# can be one of DEBUG, INFO, WARN and ERROR, in increasing order of importance and severity. +# +# This should be at INFO or DEBUG in order to get support for draupnir problems. +logLevel: "INFO" + +# Whether or not draupnir should synchronize policy lists immediately after startup. +# Equivalent to running '!draupnir sync'. +syncOnStartup: true + +# Whether or not draupnir should check moderation permissions in all protected rooms on startup. +# Equivalent to running `!draupnir verify`. +verifyPermissionsOnStartup: true + +# Whether or not draupnir should actually apply bans and policy lists, +# turn on to trial some untrusted configuration or lists. +noop: false + +# Whether draupnir should check member lists quicker (by using a different endpoint), +# keep in mind that enabling this will miss invited (but not joined) users. +# +# Turn on if your bot is in (very) large rooms, or in large amounts of rooms. +fasterMembershipChecks: false + +# A case-insensitive list of ban reasons to have the bot also automatically redact the user's messages for. +# +# If the bot sees you ban a user with a reason that is an (exact case-insensitive) match to this list, +# it will also remove the user's messages automatically. +# +# Typically this is useful to avoid having to give two commands to the bot. +# Advanced: Use asterisks to have the reason match using "globs" +# (f.e. "spam*testing" would match "spam for testing" as well as "spamtesting"). +# +# See here for more info: https://www.digitalocean.com/community/tools/glob +# Note: Keep in mind that glob is NOT regex! +automaticallyRedactForReasons: + - "spam" + - "advertising" + +# A list of rooms to protect. draupnir will add this to the list it knows from its account data. +# +# It won't, however, add it to the account data. +# Manually add the room via '!draupnir rooms add' to have it stay protected regardless if this config value changes. +# +# Note: These must be matrix.to URLs +#protectedRooms: +# - "https://matrix.to/#/#yourroom:example.org" + +# Whether or not to add all joined rooms to the "protected rooms" list +# (excluding the management room and watched policy list rooms, see below). +# +# Note that this effectively makes the protectedRooms and associated commands useless +# for regular rooms. +# +# Note: the management room is *excluded* from this condition. +# Explicitly add it as a protected room to protect it. +# +# Note: Ban list rooms the bot is watching but didn't create will not be protected. +# Explicitly add these rooms as a protected room list if you want them protected. +protectAllJoinedRooms: false + +# Increase this delay to have Mjölnir wait longer between two consecutive backgrounded +# operations. The total duration of operations will be longer, but the homeserver won't +# be affected as much. Conversely, decrease this delay to have Mjölnir chain operations +# faster. The total duration of operations will generally be shorter, but the performance +# of the homeserver may be more impacted. +backgroundDelayMS: 500 + +# Server administration commands, these commands will only work if draupnir is +# a global server administrator, and the bot's server is a Synapse instance. +#admin: +# # Whether or not draupnir can temporarily take control of any eligible account from the local homeserver who's in the room +# # (with enough permissions) to "make" a user an admin. +# # +# # This only works if a local user with enough admin permissions is present in the room. +# enableMakeRoomAdminCommand: false + +# Misc options for command handling and commands +commands: + # Whether or not the `!draupnir` prefix is necessary to submit commands. + # + # If `true`, will allow commands like `!ban`, `!help`, etc. + # + # Note: draupnir can also be pinged by display name instead of having to use + # the !draupnir prefix. For example, "my_moderator_bot: ban @spammer:example.org" + # will address only my_moderator_bot. + allowNoPrefix: false + + # Any additional bot prefixes that draupnir will listen to. i.e. adding `mod` will allow `!mod help`. + additionalPrefixes: + - "draupnir_bot" + + # Whether or not commands with a wildcard (*) will require an additional `--force` argument + # in the command to be able to be submitted. + confirmWildcardBan: true + +# Configuration specific to certain toggle-able protections +#protections: +# # Configuration for the wordlist plugin, which can ban users based if they say certain +# # blocked words shortly after joining. +# wordlist: +# # A list of case-insensitive keywords that the WordList protection will watch for from new users. +# # +# # WordList will ban users who use these words when first joining a room, so take caution when selecting them. +# # +# # For advanced usage, regex can also be used, see the following links for more information; +# # - https://www.digitalocean.com/community/tutorials/an-introduction-to-regular-expressions +# # - https://regexr.com/ +# # - https://regexone.com/ +# words: +# - "LoReM" +# - "IpSuM" +# - "DoLoR" +# - "aMeT" +# +# # For how long (in minutes) the user is "new" to the WordList plugin. +# # +# # After this time, the user will no longer be banned for using a word in the above wordlist. +# # +# # Set to zero to disable the timeout and make users *always* appear "new". +# # (users will always be banned if they say a bad word) +# minutesBeforeTrusting: 20 + +# Options for advanced monitoring of the health of the bot. +health: + # healthz options. These options are best for use in container environments + # like Kubernetes to detect how healthy the service is. The bot will report + # that it is unhealthy until it is able to process user requests. Typically + # this means that it'll flag itself as unhealthy for a number of minutes + # before saying "Now monitoring rooms" and flagging itself healthy. + # + # Health is flagged through HTTP status codes, defined below. + healthz: + # Whether the healthz integration should be enabled (default false) + enabled: false + + # The port to expose the webserver on. Defaults to 8080. + port: 8080 + + # The address to listen for requests on. Defaults to all addresses. + address: "0.0.0.0" + + # The path to expose the monitoring endpoint at. Defaults to `/healthz` + endpoint: "/healthz" + + # The HTTP status code which reports that the bot is healthy/ready to + # process requests. Typically this should not be changed. Defaults to + # 200. + healthyStatus: 200 + + # The HTTP status code which reports that the bot is not healthy/ready. + # Defaults to 418. + unhealthyStatus: 418 + +# Options for exposing web APIs. +#web: +# # Whether to enable web APIs. +# enabled: false +# +# # The port to expose the webserver on. Defaults to 8080. +# port: 8080 +# +# # The address to listen for requests on. Defaults to only the current +# # computer. +# address: localhost +# +# # Alternative setting to open to the entire web. Be careful, +# # as this will increase your security perimeter: +# # +# # address: "0.0.0.0" +# +# # A web API designed to intercept Matrix API +# # POST /_matrix/client/r0/rooms/{roomId}/report/{eventId} +# # and display readable abuse reports in the moderation room. +# # +# # If you wish to take advantage of this feature, you will need +# # to configure a reverse proxy, see e.g. test/nginx.conf +# abuseReporting: +# # Whether to enable this feature. +# enabled: false + +# Whether or not to actively poll synapse for abuse reports, to be used +# instead of intercepting client calls to synapse's abuse endpoint, when that +# isn't possible/practical. +pollReports: false + +# Whether or not new reports, received either by webapi or polling, +# should be printed to our managementRoom. +displayReports: false diff --git a/roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-mjolnir.service.j2 b/roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-mjolnir.service.j2 new file mode 100644 index 00000000..6995bcc3 --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-mjolnir.service.j2 @@ -0,0 +1,42 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Draupnir bot +{% for service in matrix_bot_draupnir_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_bot_draupnir_systemd_required_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-draupnir 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-draupnir 2>/dev/null || true' + +# Intentional delay, so that the homeserver (we likely depend on) can manage to start. +ExecStartPre={{ matrix_host_command_sleep }} 5 + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-bot-draupnir \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --network={{ matrix_docker_network }} \ + --mount type=bind,src={{ matrix_bot_draupnir_config_path }},dst=/data/config,ro \ + --mount type=bind,src={{ matrix_bot_draupnir_data_path }},dst=/data \ + {% for arg in matrix_bot_draupnir_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_bot_draupnir_docker_image }} + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-draupnir 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-draupnir 2>/dev/null || true' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-bot-draupnir + +[Install] +WantedBy=multi-user.target From 78b1ebd5af15d4acdc3e69a9cf3539cb3452a49c Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Wed, 8 Feb 2023 16:44:30 +0100 Subject: [PATCH 083/382] commit main.yml for draupnir and set target ver to develop --- .../matrix-bot-draupnir/defaults/main.yml | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 roles/custom/matrix-bot-draupnir/defaults/main.yml diff --git a/roles/custom/matrix-bot-draupnir/defaults/main.yml b/roles/custom/matrix-bot-draupnir/defaults/main.yml new file mode 100644 index 00000000..cafba72d --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/defaults/main.yml @@ -0,0 +1,59 @@ +--- +# A moderation tool for Matrix +# Project source code URL: https://github.com/Gnuxie/Draupnir + +matrix_bot_draupnir_enabled: true + +matrix_bot_draupnir_version: "develop" + +matrix_bot_draupnir_container_image_self_build: false +matrix_bot_draupnir_container_image_self_build_repo: "https://github.com/Gnuxie/Draupnir.git" + +matrix_bot_draupnir_docker_image: "{{ matrix_bot_draupnir_docker_image_name_prefix }}gnuxie/draupnir:{{ matrix_bot_draupnir_version }}" +matrix_bot_draupnir_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_draupnir_container_image_self_build else matrix_container_global_registry_prefix }}" +matrix_bot_draupnir_docker_image_force_pull: "{{ matrix_bot_draupnir_docker_image.endswith(':latest') }}" + +matrix_bot_draupnir_base_path: "{{ matrix_base_data_path }}/draupnir" +matrix_bot_draupnir_config_path: "{{ matrix_bot_draupnir_base_path }}/config" +matrix_bot_draupnir_data_path: "{{ matrix_bot_draupnir_base_path }}/data" +matrix_bot_draupnir_docker_src_files_path: "{{ matrix_bot_draupnir_base_path }}/docker-src" + +# A list of extra arguments to pass to the container +matrix_bot_draupnir_container_extra_arguments: [] + +# List of systemd services that matrix-bot-draupnir.service depends on +matrix_bot_draupnir_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-bot-draupnir.service wants +matrix_bot_draupnir_systemd_wanted_services_list: [] + +# The access token for the bot user +matrix_bot_draupnir_access_token: "" + +# The room ID where people can use the bot. The bot has no access controls, so +# anyone in this room can use the bot - secure your room! +# This should be a room alias or room ID - not a matrix.to URL. +# Note: draupnir is fairly verbose - expect a lot of messages from it. +matrix_bot_draupnir_management_room: "" + +# Default configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_bot_draupnir_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_bot_draupnir_configuration_yaml: "{{ lookup('template', 'templates/production.yaml.j2') }}" + +matrix_bot_draupnir_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_bot_draupnir_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_bot_draupnir_configuration_yaml`. + +matrix_bot_draupnir_configuration_extension: "{{ matrix_bot_draupnir_configuration_extension_yaml | from_yaml if matrix_bot_draupnir_configuration_extension_yaml | from_yaml is mapping else {} }}" + +# Holds the final configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_bot_draupnir_configuration_yaml`. +matrix_bot_draupnir_configuration: "{{ matrix_bot_draupnir_configuration_yaml | from_yaml | combine(matrix_bot_draupnir_configuration_extension, recursive=True) }}" From 9092d4bb6b6348b9c7ad068f308f52392ba0e104 Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Wed, 8 Feb 2023 17:02:59 +0100 Subject: [PATCH 084/382] Push draupnir version from develop to v1.80.0-beta.0 --- roles/custom/matrix-bot-draupnir/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-draupnir/defaults/main.yml b/roles/custom/matrix-bot-draupnir/defaults/main.yml index cafba72d..f0aa4da8 100644 --- a/roles/custom/matrix-bot-draupnir/defaults/main.yml +++ b/roles/custom/matrix-bot-draupnir/defaults/main.yml @@ -4,7 +4,7 @@ matrix_bot_draupnir_enabled: true -matrix_bot_draupnir_version: "develop" +matrix_bot_draupnir_version: "v1.80.0-beta.0" matrix_bot_draupnir_container_image_self_build: false matrix_bot_draupnir_container_image_self_build_repo: "https://github.com/Gnuxie/Draupnir.git" From 1338963b6cac0273ef078983bac3851fbeedad51 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 8 Feb 2023 18:47:19 +0200 Subject: [PATCH 085/382] Add support for obtaining additional SSL certificates via Traefik --- group_vars/matrix_servers | 2 ++ requirements.yml | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index d146b252..fbc83fa1 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3443,6 +3443,8 @@ devture_traefik_additional_entrypoints_auto: host_bind_port: "{{ matrix_federation_public_port }}" config: {} +devture_traefik_additional_domains_to_obtain_certificates_for: "{{ matrix_ssl_additional_domains_to_obtain_certificates_for }}" + ######################################################################## # # # /com.devture.ansible.role.traefik # diff --git a/requirements.yml b/requirements.yml index b0dd8d1a..1edda46a 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: c90a0adcdc1de3d00d256e794b095ffbf466bb05 + version: 650adfa75931714d9bc7c596cab71e33a4873a4e - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 From c71567477a0fb762dae6e56938c682b21e25e99e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 8 Feb 2023 18:48:01 +0200 Subject: [PATCH 086/382] Stop using deprecated matrix_bot_postmoogle_domain variable in group vars --- group_vars/matrix_servers | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6d096dea..162d6ccc 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2306,7 +2306,7 @@ matrix_ssl_domains_to_obtain_certificates_for: | + ([matrix_server_fqn_ntfy] if matrix_ntfy_enabled else []) + - ([matrix_bot_postmoogle_domain] if matrix_bot_postmoogle_enabled else []) + (matrix_bot_postmoogle_domains if matrix_bot_postmoogle_enabled else []) + ([matrix_domain] if matrix_nginx_proxy_base_domain_serving_enabled else []) + From 7b42ff4b759afc031cce73218d7199573483e87c Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Wed, 8 Feb 2023 18:55:08 +0100 Subject: [PATCH 087/382] Finalise moving draupnir to a fully testable state. --- group_vars/matrix_servers | 30 ++++++++++++++++++++++++++++++ playbooks/matrix.yml | 1 + 2 files changed, 31 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6d096dea..1c0a14e4 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -189,6 +189,8 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-bot-mjolnir.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'mjolnir']}] if matrix_bot_mjolnir_enabled else []) + + ([{'name': 'matrix-bot-draupnir.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'draupnir']}] if matrix_bot_draupnir_enabled else []) + + ([{'name': 'matrix-bot-postmoogle.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'postmoogle']}] if matrix_bot_postmoogle_enabled else []) + ([{'name': 'matrix-bot-chatgpt.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'chatgpt']}] if matrix_bot_chatgpt_enabled else []) @@ -1759,6 +1761,34 @@ matrix_bot_mjolnir_systemd_required_services_list: | # ###################################################################### +###################################################################### +# +# matrix-bot-draupnir +# +###################################################################### + +# We don't enable bots by default. +matrix_bot_draupnir_enabled: false + +matrix_bot_draupnir_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" + +matrix_bot_draupnir_systemd_required_services_list: | + {{ + ['docker.service'] + + + ['matrix-' + matrix_homeserver_implementation + '.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + + + (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) + }} + +###################################################################### +# +# /matrix-bot-draupnir +# +###################################################################### + ###################################################################### # # matrix-backup-borg diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 7be0d629..90678454 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -79,6 +79,7 @@ - custom/matrix-bot-postmoogle - custom/matrix-bot-go-neb - custom/matrix-bot-mjolnir + - custom/matrix-bot-draupnir - custom/matrix-bot-chatgpt - custom/matrix-cactus-comments - custom/matrix-synapse From 79a4e57fbdf9d900e3753bc0cea595c2a5e02bfc Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Wed, 8 Feb 2023 18:55:27 +0100 Subject: [PATCH 088/382] Add draupnir docs. --- docs/configuring-playbook-bot-draupnir.md | 114 ++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 docs/configuring-playbook-bot-draupnir.md diff --git a/docs/configuring-playbook-bot-draupnir.md b/docs/configuring-playbook-bot-draupnir.md new file mode 100644 index 00000000..cbec1c4f --- /dev/null +++ b/docs/configuring-playbook-bot-draupnir.md @@ -0,0 +1,114 @@ +# Setting up draupnir (optional) + +The playbook can install and configure the [draupnir](https://github.com/Gnuxie/Draupnir) moderation bot for you. + +See the project's [documentation](https://github.com/Gnuxie/Draupnir) to learn what it does and why it might be useful to you. + +If your migrating from Mjolnir skip to step 5b. + +## 1. Register the bot account + +The playbook does not automatically create users for you. The bot requires an access token to be able to connect to your homeserver. + +You **need to register the bot user manually** before setting up the bot. + +Choose a strong password for the bot. You can generate a good password with a command like this: `pwgen -s 64 1`. + +You can use the playbook to [register a new user](registering-users.md): + +``` +ansible-playbook -i inventory/hosts setup.yml --extra-vars='username=bot.draupnir password=PASSWORD_FOR_THE_BOT admin=no' --tags=register-user +``` + +If you would like draupnir to be able to deactivate users, move aliases, shutdown rooms, etc then it must be a server admin so you need to change `admin=no` to `admin=yes` in the command above. + + +## 2. Get an access token + +Refer to the documentation on [how to obtain an access token](obtaining-access-tokens.md). + + +## 3. Make sure the account is free from rate limiting + +You will need to prevent Synapse from rate limiting the bot's account. This is not an optional step. If you do not do this step draupnir will crash. This can be done using Synapse's [admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#override-ratelimiting-for-users). This can also be manually done by editing the Synapse database. Manually editing the Synapse database is rarely a good idea. Please ask for help if you are uncomfortable with these steps. + +1. Copy the statement below into a text editor. + + ``` + INSERT INTO ratelimit_override VALUES ('@bot.draupnir:DOMAIN', 0, 0); + ``` + +1. Change the username (`@bot.draupnir:DOMAIN`) to the username you used when you registered the bot's account. You must change `DOMAIN` to your server's domain. + +1. Get a database terminal by following these steps: [maintenance-postgres.md#getting-a-database-terminal](maintenance-postgres.md#getting-a-database-terminal) + +1. Connect to Synapse's database by typing `\connect synapse` into the database terminal + +1. Paste in the `INSERT INTO` command that you edited and press enter. + +You can run `SELECT * FROM ratelimit_override;` to see if it worked. If the output looks like this: + +``` + user_id | messages_per_second | burst_count +-----------------------+---------------------+------------- + @bot.draupnir:raim.ist | 0 | 0` +``` +then you did it correctly. + + +## 4. Create a management room + +Using your own account, create a new invite only room that you will use to manage the bot. This is the room where you will see the status of the bot and where you will send commands to the bot, such as the command to ban a user from another room. Anyone in this room can control the bot so it is important that you only invite trusted users to this room. The room must be unencrypted since the playbook does not support installing Pantalaimon yet. + +Once you have created the room you need to copy the room ID so you can tell the bot to use that room. In Element you can do this by going to the room's settings, clicking Advanced, and then coping the internal room ID. The room ID will look something like `!QvgVuKq0ha8glOLGMG:DOMAIN`. + +Finally invite the `@bot.draupnir:DOMAIN` account you created earlier into the room. + + +## 5a. Adjusting the playbook configuration + +Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file (adapt to your needs): + +You must replace `ACCESS_TOKEN_FROM_STEP_2_GOES_HERE` and `ROOM_ID_FROM_STEP_4_GOES_HERE` with the your own values. + +```yaml +matrix_bot_draupnir_enabled: true + +matrix_bot_draupnir_access_token: "ACCESS_TOKEN_FROM_STEP_2_GOES_HERE" + +matrix_bot_draupnir_management_room: "ROOM_ID_FROM_STEP_4_GOES_HERE" +``` + +## 5b. Migrating from Mjolnir (Only required if migrating.) + +Replace your matrix_bot_mjolnir config with matrix_bot_draupnir config. Also disable mjolnir if your doing migration. +That is all you need to do due to that Draupnir can complete migration on its own. + +## 6. Installing + +After configuring the playbook, run the [installation](installing.md) command: + +``` +ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start +``` + + +## Usage + +You can refer to the upstream [documentation](https://github.com/Gnuxie/Draupnir) for additional ways to use and configure draupnir. Check out their [quickstart guide](https://github.com/matrix-org/draupnir/blob/main/docs/moderators.md#quick-usage) for some basic commands you can give to the bot. + +You can configure additional options by adding the `matrix_bot_draupnir_configuration_extension_yaml` variable to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file. + +For example to change draupnir's `recordIgnoredInvites` option to `true` you would add the following to your `vars.yml` file. + +```yaml +matrix_bot_draupnir_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_bot_draupnir_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_bot_draupnir_configuration_yaml`. + recordIgnoredInvites: true +``` From a717590aa5e00e988d1c6db0b72b45a0decf3c31 Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Wed, 8 Feb 2023 19:53:35 +0100 Subject: [PATCH 089/382] Rename systemd service file from mjolnir to draupnir --- ...trix-bot-mjolnir.service.j2 => matrix-bot-draupnir.service.j2} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename roles/custom/matrix-bot-draupnir/templates/systemd/{matrix-bot-mjolnir.service.j2 => matrix-bot-draupnir.service.j2} (100%) diff --git a/roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-mjolnir.service.j2 b/roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-draupnir.service.j2 similarity index 100% rename from roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-mjolnir.service.j2 rename to roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-draupnir.service.j2 From ddcb1735e2671344dec7742b6d5726edfaeeecba Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Wed, 8 Feb 2023 20:05:47 +0100 Subject: [PATCH 090/382] Add draupnir as valid prefix to resolve a bug Current draupnir does not listen to its name. This config change fixes this bug. This bodge is able to be removed once this is fixed upstream. --- roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 | 1 + 1 file changed, 1 insertion(+) diff --git a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 index 06d88f48..dcc61b1b 100644 --- a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 +++ b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 @@ -146,6 +146,7 @@ commands: # Any additional bot prefixes that draupnir will listen to. i.e. adding `mod` will allow `!mod help`. additionalPrefixes: - "draupnir_bot" + - "draupnir" # Whether or not commands with a wildcard (*) will require an additional `--force` argument # in the command to be able to be submitted. From a5683a64497e3275908bdeabd3e590b5c61320ef Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 9 Feb 2023 10:12:09 +0200 Subject: [PATCH 091/382] Upgrade com.devture.ansible.role.traefik and rename some variables --- group_vars/matrix_servers | 6 +++--- requirements.yml | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 88800113..fcc9c84e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2300,7 +2300,7 @@ matrix_nginx_proxy_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" matrix_nginx_proxy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" -matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_config_entrypoint_primary }}" +matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled: "{{ matrix_client_element_enabled }}" @@ -3102,7 +3102,7 @@ prometheus_node_exporter_container_additional_networks: "{{ [matrix_playbook_rev prometheus_node_exporter_container_labels_traefik_enabled: false prometheus_node_exporter_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" -prometheus_node_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_config_entrypoint_primary }}" +prometheus_node_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" ###################################################################### # @@ -3133,7 +3133,7 @@ prometheus_postgres_exporter_container_additional_networks: "{{ [matrix_playbook prometheus_postgres_exporter_container_labels_traefik_enabled: false prometheus_postgres_exporter_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" -prometheus_postgres_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_config_entrypoint_primary }}" +prometheus_postgres_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" prometheus_postgres_exporter_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" prometheus_postgres_exporter_database_username: matrix_prometheus_postgres_exporter diff --git a/requirements.yml b/requirements.yml index 1edda46a..aa1c14de 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: 650adfa75931714d9bc7c596cab71e33a4873a4e + version: b52ecc4df030fb99ca547575126baaf916db0743 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 From c289996cd9a72281263a095fffbc7b95de60bb9e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 9 Feb 2023 10:16:41 +0200 Subject: [PATCH 092/382] Upgrade com.devture.ansible.role.traefik --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index aa1c14de..1341bbbe 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: b52ecc4df030fb99ca547575126baaf916db0743 + version: fb09fd26f877372417d5586f1e79e83f983f0bd6 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 From 9615855cfa7e86bb3843f5fa2db90a9b3b9a5fc4 Mon Sep 17 00:00:00 2001 From: ntallasv Date: Thu, 9 Feb 2023 14:53:56 +0200 Subject: [PATCH 093/382] update matrix-chatgpt-bot --- roles/custom/matrix-bot-chatgpt/defaults/main.yml | 13 +++++++------ .../matrix-bot-chatgpt/tasks/validate_config.yml | 4 +--- roles/custom/matrix-bot-chatgpt/templates/env.j2 | 11 ++++++----- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index 324e64ab..489b22c4 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_chatgpt_docker_repo: "https://github.com/matrixgpt/matrix-chatgpt-bot matrix_bot_chatgpt_docker_repo_version: "{{ 'latest' if matrix_bot_chatgpt_version == 'latest' else matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_src_files_path: "{{ matrix_base_data_path }}/chatgpt/docker-src" -matrix_bot_chatgpt_version: 1.4.1 +matrix_bot_chatgpt_version: 2.2.1 matrix_bot_chatgpt_docker_image: "{{ matrix_bot_chatgpt_docker_image_name_prefix }}matrixgpt/matrix-chatgpt-bot:{{ matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_chatgpt_container_image_self_build else 'ghcr.io/' }}" matrix_bot_chatgpt_docker_image_force_pull: "{{ matrix_bot_chatgpt_docker_image.endswith(':latest') }}" @@ -28,13 +28,14 @@ matrix_bot_chatgpt_systemd_required_services_list: ['docker.service'] matrix_bot_chatgpt_systemd_wanted_services_list: [] # ChatGPT Settings -matrix_bot_chatgpt_openai_email: '' # OPENAI_EMAIL= -matrix_bot_chatgpt_openai_password: '' # OPENAI_PASSWORD= -matrix_bot_chatgpt_openai_login_type: 'google' # OPENAI_LOGIN_TYPE="google" -matrix_bot_chatgpt_openai_pro: false # OPENAI_PRO - +matrix_bot_chatgpt_openai_api_key: '' matrix_bot_chatgpt_context: 'thread' # CHATGPT_CONTEXT="thread" +matrix_bot_chatgpt_keyv_backend: 'file' +matrix_bot_chatgpt_keyv_url: '' +matrix_bot_chatgpt_keyv_bot_encryption: false +matrix_bot_chatgpt_keyv_bot_storage: true + # Matrix Static Settings (required, see notes) # Defaults to "https://matrix.org" matrix_bot_chatgpt_matrix_homeserver_url: "{{ matrix_homeserver_container_url }}" # MATRIX_HOMESERVER_URL= diff --git a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml index f68d0e5c..3d30994c 100644 --- a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml +++ b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml @@ -6,7 +6,5 @@ You need to define a required configuration setting (`{{ item.name }}`). when: "item.when | bool and vars[item.name] == ''" with_items: - - {'name': 'matrix_bot_chatgpt_openai_email', when: true} - - {'name': 'matrix_bot_chatgpt_openai_password', when: true} - - {'name': 'matrix_bot_chatgpt_openai_login_type', when: true} + - {'name': 'matrix_bot_chatgpt_openai_api_key', when: true} - {'name': 'matrix_bot_chatgpt_matrix_bot_username', when: true} diff --git a/roles/custom/matrix-bot-chatgpt/templates/env.j2 b/roles/custom/matrix-bot-chatgpt/templates/env.j2 index faf6be71..40b2f9ff 100644 --- a/roles/custom/matrix-bot-chatgpt/templates/env.j2 +++ b/roles/custom/matrix-bot-chatgpt/templates/env.j2 @@ -1,13 +1,14 @@ MATRIX_HOMESERVER_URL={{ matrix_bot_chatgpt_matrix_homeserver_url }} MATRIX_ACCESS_TOKEN={{ matrix_bot_chatgpt_matrix_access_token }} -OPENAI_EMAIL={{ matrix_bot_chatgpt_openai_email }} -OPENAI_PASSWORD={{ matrix_bot_chatgpt_openai_password }} -OPENAI_LOGIN_TYPE={{ matrix_bot_chatgpt_openai_login_type }} -OPENAI_PRO={{ matrix_bot_chatgpt_openai_pro | to_json }} - +OPENAI_API_KEY={{ matrix_bot_chatgpt_openai_api_key }} CHATGPT_CONTEXT={{ matrix_bot_chatgpt_context }} +KEYV_BACKEND={{ matrix_bot_chatgpt_keyv_backend }} +KEYV_URL={{ matrix_bot_chatgpt_keyv_url }} +KEYV_BOT_ENCRYPTION={{ matrix_bot_chatgpt_keyv_bot_encryption|lower }} +KEYV_BOT_STORAGE={{ matrix_bot_chatgpt_keyv_bot_storage|lower }} + # With the @ and :DOMAIN, ie @SOMETHING:DOMAIN MATRIX_BOT_USERNAME={{ matrix_bot_chatgpt_matrix_bot_username }} MATRIX_BOT_PASSWORD={{ matrix_bot_chatgpt_matrix_bot_password }} From d32f80bf29c8fe3750534e0bf14970c3e1160576 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Thu, 9 Feb 2023 17:43:35 +0000 Subject: [PATCH 094/382] Update postmoogle 0.9.11 -> 0.9.12 * fix uploads from incoming emails into matrix threads * fix emails dequeue (account data cleanup) * rewrite recipients handling (Cc, To, etc.) --- roles/custom/matrix-bot-postmoogle/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-postmoogle/defaults/main.yml b/roles/custom/matrix-bot-postmoogle/defaults/main.yml index 48a2b45c..9f3dad24 100644 --- a/roles/custom/matrix-bot-postmoogle/defaults/main.yml +++ b/roles/custom/matrix-bot-postmoogle/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_postmoogle_docker_repo: "https://gitlab.com/etke.cc/postmoogle.git" matrix_bot_postmoogle_docker_repo_version: "{{ 'main' if matrix_bot_postmoogle_version == 'latest' else matrix_bot_postmoogle_version }}" matrix_bot_postmoogle_docker_src_files_path: "{{ matrix_base_data_path }}/postmoogle/docker-src" -matrix_bot_postmoogle_version: v0.9.11 +matrix_bot_postmoogle_version: v0.9.12 matrix_bot_postmoogle_docker_image: "{{ matrix_bot_postmoogle_docker_image_name_prefix }}etke.cc/postmoogle:{{ matrix_bot_postmoogle_version }}" matrix_bot_postmoogle_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_postmoogle_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_postmoogle_docker_image_force_pull: "{{ matrix_bot_postmoogle_docker_image.endswith(':latest') }}" From b7384866844545756ee356ab0a7706b1a6196891 Mon Sep 17 00:00:00 2001 From: ntallasv Date: Fri, 10 Feb 2023 00:13:31 +0200 Subject: [PATCH 095/382] update validate_config.yml --- .../matrix-bot-chatgpt/tasks/validate_config.yml | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml index 3d30994c..fcf77a8e 100644 --- a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml +++ b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml @@ -6,5 +6,17 @@ You need to define a required configuration setting (`{{ item.name }}`). when: "item.when | bool and vars[item.name] == ''" with_items: - - {'name': 'matrix_bot_chatgpt_openai_api_key', when: true} - - {'name': 'matrix_bot_chatgpt_matrix_bot_username', when: true} + - { 'name': 'matrix_bot_chatgpt_openai_api_key', when: true } + - { 'name': 'matrix_bot_chatgpt_matrix_bot_username', when: true } + + +- name: Fail if OpenAI configuration not up-to-date. + ansible.builtin.fail: + msg: >- + Your configuration contains a varible that is no longer used. + Please change your configuration to remove the variable (`{{ item.name }}`). + when: "item.name in vars" + with_items: + - { 'name': 'matrix_bot_chatgpt_openai_email' } + - { 'name': 'matrix_bot_chatgpt_openai_password' } + - { 'name': 'matrix_bot_chatgpt_openai_login_type' } From 4221b64a9805a8c9da3a3dc83b5f87a0bf9fe0f3 Mon Sep 17 00:00:00 2001 From: ntallasv Date: Fri, 10 Feb 2023 00:14:09 +0200 Subject: [PATCH 096/382] update configuring-playbook-bot-chatgpt.md --- docs/configuring-playbook-bot-chatgpt.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/configuring-playbook-bot-chatgpt.md b/docs/configuring-playbook-bot-chatgpt.md index eb0c7596..7a006816 100644 --- a/docs/configuring-playbook-bot-chatgpt.md +++ b/docs/configuring-playbook-bot-chatgpt.md @@ -32,11 +32,8 @@ Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars. ```yaml matrix_bot_chatgpt_enabled: true -# See instructions on -# https://www.npmjs.com/package/chatgpt -matrix_bot_chatgpt_openai_email: '' -matrix_bot_chatgpt_openai_password: '' -matrix_bot_chatgpt_openai_login_type: google +# Obtain a new API key from https://platform.openai.com/account/api-keys +matrix_bot_chatgpt_openai_api_key: '' # This is the default username # matrix_bot_chatgpt_matrix_bot_username_localpart: 'bot.chatgpt' From f71cd3a760657bff3b89ebf172f83c26f94e5834 Mon Sep 17 00:00:00 2001 From: ntallasv Date: Fri, 10 Feb 2023 00:34:07 +0200 Subject: [PATCH 097/382] fix linting in validate_config.yml --- .../matrix-bot-chatgpt/tasks/validate_config.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml index fcf77a8e..ecc2b8d2 100644 --- a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml +++ b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml @@ -6,8 +6,8 @@ You need to define a required configuration setting (`{{ item.name }}`). when: "item.when | bool and vars[item.name] == ''" with_items: - - { 'name': 'matrix_bot_chatgpt_openai_api_key', when: true } - - { 'name': 'matrix_bot_chatgpt_matrix_bot_username', when: true } + - {'name': 'matrix_bot_chatgpt_openai_api_key', when: true} + - {'name': 'matrix_bot_chatgpt_matrix_bot_username', when: true} - name: Fail if OpenAI configuration not up-to-date. @@ -17,6 +17,6 @@ Please change your configuration to remove the variable (`{{ item.name }}`). when: "item.name in vars" with_items: - - { 'name': 'matrix_bot_chatgpt_openai_email' } - - { 'name': 'matrix_bot_chatgpt_openai_password' } - - { 'name': 'matrix_bot_chatgpt_openai_login_type' } + - {'name': 'matrix_bot_chatgpt_openai_email'} + - {'name': 'matrix_bot_chatgpt_openai_password'} + - {'name': 'matrix_bot_chatgpt_openai_login_type'} From 0a15e56641b81801fc72c8aab7da0ca4282447fe Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 11:41:44 +0200 Subject: [PATCH 098/382] Fix typo --- docs/configuring-playbook-bot-draupnir.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-bot-draupnir.md b/docs/configuring-playbook-bot-draupnir.md index cbec1c4f..7626db24 100644 --- a/docs/configuring-playbook-bot-draupnir.md +++ b/docs/configuring-playbook-bot-draupnir.md @@ -81,7 +81,7 @@ matrix_bot_draupnir_management_room: "ROOM_ID_FROM_STEP_4_GOES_HERE" ## 5b. Migrating from Mjolnir (Only required if migrating.) -Replace your matrix_bot_mjolnir config with matrix_bot_draupnir config. Also disable mjolnir if your doing migration. +Replace your matrix_bot_mjolnir config with matrix_bot_draupnir config. Also disable mjolnir if you're doing migration. That is all you need to do due to that Draupnir can complete migration on its own. ## 6. Installing From 600aaa0bc7db67eff0777d3707f199461d80e749 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 11:52:25 +0200 Subject: [PATCH 099/382] Announce Draupnir support Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2451 --- CHANGELOG.md | 9 +++++++++ docs/configuring-playbook-bot-draupnir.md | 6 +++--- docs/configuring-playbook.md | 2 ++ 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bb62d8dd..69e932a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,12 @@ +# 2023-02-10 + +## Draupnir moderation tool (bot) support + +Thanks to [FSG-Cat](https://github.com/FSG-Cat), the playbook can now install and configure the [Draupnir](https://github.com/Gnuxie/Draupnir) moderation tool (bot). Draupnir is a fork of [Mjolnir](docs/configuring-playbook-bot-mjolnir.md) (which the playbook has supported for a long time) maintained by Mjolnir's former lead developer. + +Additional details are available in [Setting up Draupnir](docs/configuring-playbook-bot-draupnir.md). + + # 2023-02-05 ## The matrix-prometheus-postgres-exporter role lives independently now diff --git a/docs/configuring-playbook-bot-draupnir.md b/docs/configuring-playbook-bot-draupnir.md index 7626db24..6704ad65 100644 --- a/docs/configuring-playbook-bot-draupnir.md +++ b/docs/configuring-playbook-bot-draupnir.md @@ -32,7 +32,7 @@ Refer to the documentation on [how to obtain an access token](obtaining-access-t You will need to prevent Synapse from rate limiting the bot's account. This is not an optional step. If you do not do this step draupnir will crash. This can be done using Synapse's [admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#override-ratelimiting-for-users). This can also be manually done by editing the Synapse database. Manually editing the Synapse database is rarely a good idea. Please ask for help if you are uncomfortable with these steps. -1. Copy the statement below into a text editor. +1. Copy the statement below into a text editor. ``` INSERT INTO ratelimit_override VALUES ('@bot.draupnir:DOMAIN', 0, 0); @@ -81,8 +81,8 @@ matrix_bot_draupnir_management_room: "ROOM_ID_FROM_STEP_4_GOES_HERE" ## 5b. Migrating from Mjolnir (Only required if migrating.) -Replace your matrix_bot_mjolnir config with matrix_bot_draupnir config. Also disable mjolnir if you're doing migration. -That is all you need to do due to that Draupnir can complete migration on its own. +Replace your matrix_bot_mjolnir config with matrix_bot_draupnir config. Also disable mjolnir if you're doing migration. +That is all you need to do due to that Draupnir can complete migration on its own. ## 6. Installing diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index b2724598..9ab50763 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -177,6 +177,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up Mjolnir](configuring-playbook-bot-mjolnir.md) - a moderation tool/bot (optional) +- [Setting up Draupnir](configuring-playbook-bot-draupnir.md) - a moderation tool/bot (optional), forked from Mjolnir and maintained by its former leader developer + - [Setting up Buscarron](configuring-playbook-bot-buscarron.md) - a bot you can use to send any form (HTTP POST, HTML) to a (encrypted) Matrix room (optional) From 43a6a035a046a95bfaa289bf18b9ecd76c1521db Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 14:16:00 +0200 Subject: [PATCH 100/382] Skip removing /.well-known/element directory to suppress ansible-lint error Leaving an orphan directory is okay and can be improved later on. --- .../tasks/util/setup_jitsi_auth_uvs_uninstall.yml | 8 -------- 1 file changed, 8 deletions(-) diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml index 229ce896..5c742566 100644 --- a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml @@ -10,14 +10,6 @@ - "{{ matrix_jitsi_prosody_auth_matrix_files | map(attribute='path') | map('regex_replace', '^', matrix_jitsi_prosody_plugins_path+'/') | list }}" register: matrix_jitsi_prosody_auth_matrix_user_verification_uninstalled -- name: Remove .well-known/element directory if empty - ansible.builtin.command: - argv: - - rmdir - - "{{ matrix_static_files_base_path }}/.well-known/element" - removes: "{{matrix_static_files_base_path}}/.well-known/element" - ignore_errors: true - - when: matrix_jitsi_prosody_auth_matrix_user_verification_uninstalled.changed block: - name: Populate service facts From e1bfa2a7d632735410d9682e00c8a4ea1f953ba5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 14:21:31 +0200 Subject: [PATCH 101/382] Fix ansible-lint-reported errors --- .../custom/matrix-jitsi/tasks/self_check_matrix_auth.yml | 1 + .../tasks/util/setup_jitsi_auth_uvs_uninstall.yml | 2 +- .../tasks/util/setup_jitsi_prosody_post_setup_hooks.yml | 4 ++-- .../matrix-user-verification-service/handlers/main.yml | 2 +- .../tasks/setup_uninstall.yml | 8 ++++---- .../tasks/validate_config.yml | 8 ++++---- 6 files changed, 13 insertions(+), 12 deletions(-) diff --git a/roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml b/roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml index 68a28ef7..52d865c5 100644 --- a/roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml +++ b/roles/custom/matrix-jitsi/tasks/self_check_matrix_auth.yml @@ -48,6 +48,7 @@ - "--quiet" - "{{ matrix_jitsi_prosody_self_check_uvs_health_url | quote }}" register: matrix_jitsi_prosody_self_check_uvs_result + changed_when: false ignore_errors: true - name: Fail if user verification service is not (reachable and healthy) diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml index 5c742566..2559c7e6 100644 --- a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_uninstall.yml @@ -16,5 +16,5 @@ ansible.builtin.service_facts: - name: Ensure prosody is restarted later on if currently running - set_fact: + ansible.builtin.set_fact: matrix_jitsi_prosody_require_restart: "{{ true if ansible_facts.services['matrix-jitsi-prosody.service']['state'] == 'running' else false }}" diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml index 69c18ab3..6be6092a 100644 --- a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_prosody_post_setup_hooks.yml @@ -16,13 +16,13 @@ - name: Ensure matrix-jitsi-prosody container is running ansible.builtin.systemd: - state: "{{ 'restarted' if matrix_jitsi_prosody_require_restart | d(false) | bool else 'started' }}" + state: "{{ 'restarted' if matrix_jitsi_prosody_require_restart | default(false) | bool else 'started' }}" name: matrix-jitsi-prosody register: matrix_jitsi_prosody_start_result # If the flag was set, we can safely disable now. - name: Disable require restart flag - set_fact: + ansible.builtin.set_fact: matrix_jitsi_prosody_require_restart: false # diff --git a/roles/custom/matrix-user-verification-service/handlers/main.yml b/roles/custom/matrix-user-verification-service/handlers/main.yml index b9ee66d7..b955df80 100644 --- a/roles/custom/matrix-user-verification-service/handlers/main.yml +++ b/roles/custom/matrix-user-verification-service/handlers/main.yml @@ -1,6 +1,6 @@ --- - name: Ensure systemd reloaded after matrix-user-verification-service.service installation - service: + ansible.builtin.service: daemon_reload: true listen: "reload matrix-user-verification-service" diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml index 172bf186..b61f9cdd 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_uninstall.yml @@ -1,25 +1,25 @@ --- - name: Check existence of matrix-user-verification-service service - stat: + ansible.builtin.stat: path: "{{ devture_systemd_docker_base_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" register: matrix_user_verification_service_service_stat - when: matrix_user_verification_service_service_stat.stat.exists | bool block: - name: Ensure matrix-user-verification-service is stopped - service: + ansible.builtin.service: name: "{{ matrix_user_verification_service_systemd_service_basename }}" state: stopped daemon_reload: true register: stopping_result - name: Ensure matrix-user-verification-service.service doesn't exist - file: + ansible.builtin.file: path: "{{ devture_systemd_docker_base_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" state: absent - name: Ensure Matrix user-verification-service paths don't exist - file: + ansible.builtin.file: path: "{{ matrix_user_verification_service_base_path }}" state: absent diff --git a/roles/custom/matrix-user-verification-service/tasks/validate_config.yml b/roles/custom/matrix-user-verification-service/tasks/validate_config.yml index 40e9090c..9d7f1be8 100644 --- a/roles/custom/matrix-user-verification-service/tasks/validate_config.yml +++ b/roles/custom/matrix-user-verification-service/tasks/validate_config.yml @@ -1,25 +1,25 @@ --- - name: Verify homeserver_url is not empty - assert: + ansible.builtin.assert: that: - matrix_user_verification_service_uvs_homeserver_url|length > 0 fail_msg: "Missing variable in {{ matrix_user_verification_service_ansible_name }} role" - name: Verify Auth is configured properly or disabled - assert: + ansible.builtin.assert: that: - matrix_user_verification_service_uvs_access_token|length > 0 or not matrix_user_verification_service_uvs_require_auth|bool fail_msg: "If Auth is enabled, a valid (non empty) TOKEN must be given in 'matrix_user_verification_service_uvs_access_token'." - name: Verify server_name for openid verification is given, if pinning a single server_name is enabled. - assert: + ansible.builtin.assert: that: - matrix_user_verification_service_uvs_openid_verify_server_name|length > 0 or not matrix_user_verification_service_uvs_pin_openid_verify_server_name|bool fail_msg: "If pinning a single server_name is enabled, a valid (non empty) server_name must be given in 'matrix_user_verification_service_uvs_openid_verify_server_name'." - name: Verify the homeserver implementation is synapse - assert: + ansible.builtin.assert: that: - matrix_homeserver_implementation == 'synapse' fail_msg: "The User-Verification-Service requires Synapse as homeserver implementation" From f6ab162fff21e59c3b7eb095fb180beb4fc5a309 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 14:22:37 +0200 Subject: [PATCH 102/382] Remove systemd-reloading handler in matrix-user-verification-service None of the other roles use handlers. We rely on com.devture.ansible.role.systemd_service_manager to reload services when it's necessary to do so. --- .../matrix-user-verification-service/handlers/main.yml | 6 ------ .../tasks/setup_install.yml | 1 - 2 files changed, 7 deletions(-) delete mode 100644 roles/custom/matrix-user-verification-service/handlers/main.yml diff --git a/roles/custom/matrix-user-verification-service/handlers/main.yml b/roles/custom/matrix-user-verification-service/handlers/main.yml deleted file mode 100644 index b955df80..00000000 --- a/roles/custom/matrix-user-verification-service/handlers/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- - -- name: Ensure systemd reloaded after matrix-user-verification-service.service installation - ansible.builtin.service: - daemon_reload: true - listen: "reload matrix-user-verification-service" diff --git a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml index d095c410..185b4b64 100644 --- a/roles/custom/matrix-user-verification-service/tasks/setup_install.yml +++ b/roles/custom/matrix-user-verification-service/tasks/setup_install.yml @@ -35,4 +35,3 @@ src: "{{ role_path }}/templates/systemd/matrix-user-verification-service.service.j2" dest: "{{ devture_systemd_docker_base_systemd_path }}/{{ matrix_user_verification_service_systemd_service_name }}" mode: 0644 - notify: "reload matrix-user-verification-service" From 6db3c8f2ad4c81f2b975d8913c104f73568b77d0 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 14:26:38 +0200 Subject: [PATCH 103/382] Announce Matrix Authentication Support for Jitsi Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2375 --- CHANGELOG.md | 7 +++++++ docs/configuring-playbook.md | 4 +++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 69e932a9..257f57d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # 2023-02-10 +## Matrix Authentication Support for Jitsi + +Thanks to [Jakob S.](https://github.com/jakicoll) ([zakk gGmbH](https://github.com/zakk-it)), Jitsi can now use Matrix for authentication (via [Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service)). + +Additional details are available in the [Authenticate using Matrix OpenID (Auth-Type 'matrix')](docs/configuring-playbook-jitsi.md#authenticate-using-matrix-openid-auth-type-matrix). + + ## Draupnir moderation tool (bot) support Thanks to [FSG-Cat](https://github.com/FSG-Cat), the playbook can now install and configure the [Draupnir](https://github.com/Gnuxie/Draupnir) moderation tool (bot). Draupnir is a fork of [Mjolnir](docs/configuring-playbook-bot-mjolnir.md) (which the playbook has supported for a long time) maintained by Mjolnir's former lead developer. diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index 9ab50763..3e904abc 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -99,6 +99,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up Matrix Corporal](configuring-playbook-matrix-corporal.md) (optional, advanced) +- [Matrix User Verification Service](docs/configuring-playbook-user-verification-service.md) (optional, advanced) + ### Bridging other networks @@ -177,7 +179,7 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up Mjolnir](configuring-playbook-bot-mjolnir.md) - a moderation tool/bot (optional) -- [Setting up Draupnir](configuring-playbook-bot-draupnir.md) - a moderation tool/bot (optional), forked from Mjolnir and maintained by its former leader developer +- [Setting up Draupnir](configuring-playbook-bot-draupnir.md) - a moderation tool/bot, forked from Mjolnir and maintained by its former leader developer (optional) - [Setting up Buscarron](configuring-playbook-bot-buscarron.md) - a bot you can use to send any form (HTTP POST, HTML) to a (encrypted) Matrix room (optional) From 29be262f3bda7f10d0769b2dba95fd905045e221 Mon Sep 17 00:00:00 2001 From: Catalan Lover <48515417+FSG-Cat@users.noreply.github.com> Date: Fri, 10 Feb 2023 13:34:16 +0100 Subject: [PATCH 104/382] Fix formating error in Draupnir docs --- docs/configuring-playbook-bot-draupnir.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-bot-draupnir.md b/docs/configuring-playbook-bot-draupnir.md index 6704ad65..11960474 100644 --- a/docs/configuring-playbook-bot-draupnir.md +++ b/docs/configuring-playbook-bot-draupnir.md @@ -81,7 +81,7 @@ matrix_bot_draupnir_management_room: "ROOM_ID_FROM_STEP_4_GOES_HERE" ## 5b. Migrating from Mjolnir (Only required if migrating.) -Replace your matrix_bot_mjolnir config with matrix_bot_draupnir config. Also disable mjolnir if you're doing migration. +Replace your `matrix_bot_mjolnir` config with `matrix_bot_draupnir` config. Also disable mjolnir if you're doing migration. That is all you need to do due to that Draupnir can complete migration on its own. ## 6. Installing From 28d2eb593cc0b01b4cec70a512990e6df37b9bbd Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 15:50:35 +0200 Subject: [PATCH 105/382] Add matrix_playbook_reverse_proxy_type variable which influences all other services --- group_vars/matrix_servers | 170 ++++++++++-------- roles/custom/matrix-base/defaults/main.yml | 42 +++++ .../matrix-base/tasks/validate_config.yml | 5 + .../custom/matrix-dendrite/defaults/main.yml | 2 +- 4 files changed, 147 insertions(+), 72 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index fc866f9d..62771b6c 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -30,21 +30,19 @@ matrix_playbook_docker_installation_enabled: true # In such cases, you'd like to disable the role in this playbook from bothering with Traefik at all (`matrix_playbook_traefik_role_enabled: false`). # If you used `devture_traefik_enabled: false` + `matrix_playbook_traefik_role_enabled: true` instead, you'd see the Treafik role here # try to delete Traefik data (`/devture-traefik`) installed by the other playbook. -matrix_playbook_traefik_role_enabled: false +matrix_playbook_traefik_role_enabled: true # Controls whether to attach Traefik labels to services. # This is separate from `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`, # because you may wish to disable Traefik installation by the playbook, yet still use Traefik # installed in another way. -matrix_playbook_traefik_labels_enabled: false +matrix_playbook_traefik_labels_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] }}" -matrix_playbook_traefik_certs_dumper_role_enabled: "{{ matrix_playbook_traefik_role_enabled }}" +matrix_playbook_traefik_certs_dumper_role_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] }}" # Controls the additional network that reverse-proxyable services will be connected to. matrix_playbook_reverse_proxyable_services_additional_network: "{{ devture_traefik_container_network if devture_traefik_enabled else '' }}" -matrix_playbook_reverse_proxy_type: "{{ 'traefik' if matrix_playbook_traefik_role_enabled else ('nginx-proxy' if matrix_nginx_proxy_enabled else 'none') }}" - matrix_playbook_ssl_retrieval_method: "{{ 'lets-encrypt' if matrix_playbook_traefik_certs_dumper_role_enabled else matrix_ssl_retrieval_method }}" ######################################################################## @@ -452,7 +450,7 @@ matrix_appservice_discord_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-discord over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-discord's client-server port to the local host. -matrix_appservice_discord_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9005' }}" +matrix_appservice_discord_container_http_host_bind_port: "{{ matrix_playbook_service_host_bind_interface_prefix + ':9005' if matrix_playbook_service_host_bind_interface_prefix else '' }}" # If the homeserver disables presence, it's likely better (less wasteful) to also disable presence on the bridge side. matrix_appservice_discord_bridge_disablePresence: "{{ not matrix_synapse_presence_enabled }}" @@ -499,7 +497,7 @@ matrix_appservice_webhooks_container_image_self_build: "{{ matrix_architecture ! # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-webhooks over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-webhooks' client-server port to the local host. -matrix_appservice_webhooks_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else ('127.0.0.1:' ~ matrix_appservice_webhooks_matrix_port) }}" +matrix_appservice_webhooks_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_appservice_webhooks_matrix_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_appservice_webhooks_appservice_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'webhook.as.token', rounds=655555) | to_uuid }}" @@ -538,7 +536,7 @@ matrix_appservice_slack_container_image_self_build: "{{ matrix_architecture not # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-slack over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-slack's client-server port to the local host. -matrix_appservice_slack_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else ('127.0.0.1:' ~ matrix_appservice_slack_slack_port) }}" +matrix_appservice_slack_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix + ':' ~ matrix_appservice_slack_slack_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_appservice_slack_appservice_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'slack.as.token', rounds=655555) | to_uuid }}" @@ -581,7 +579,7 @@ matrix_appservice_irc_container_image_self_build: "{{ matrix_architecture != 'am # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-irc over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-irc's client-server port to the local host. -matrix_appservice_irc_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9999' }}" +matrix_appservice_irc_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9999') if matrix_playbook_service_host_bind_interface_prefix else '' }}" # The IRC bridge docs say that if homeserver presence is disabled, it's better to also disable # IRC bridge presence, for performance reasons. @@ -837,7 +835,7 @@ matrix_mautrix_facebook_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_facebook_public_endpoint: "/{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'facebook', rounds=655555) | to_uuid }}" -matrix_mautrix_facebook_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9008' }}" +matrix_mautrix_facebook_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9008') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -888,7 +886,7 @@ matrix_mautrix_googlechat_appservice_token: "{{ '%s' | format(matrix_homeserver_ matrix_mautrix_googlechat_homeserver_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'gc.hs.token', rounds=655555) | to_uuid }}" -matrix_mautrix_googlechat_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9007' }}" +matrix_mautrix_googlechat_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9007') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_googlechat_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -930,7 +928,7 @@ matrix_mautrix_hangouts_appservice_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_hangouts_homeserver_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'ho.hs.token', rounds=655555) | to_uuid }}" -matrix_mautrix_hangouts_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9007' }}" +matrix_mautrix_hangouts_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9007') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_hangouts_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -1077,7 +1075,7 @@ matrix_mautrix_telegram_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_telegram_public_endpoint: "/{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'telegram', rounds=655555) | to_uuid }}" -matrix_mautrix_telegram_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9006' }}" +matrix_mautrix_telegram_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9006') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_telegram_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -1252,12 +1250,12 @@ matrix_hookshot_systemd_wanted_services_list: | }} matrix_hookshot_container_http_host_bind_ports_defaultmapping: - - "127.0.0.1:{{ matrix_hookshot_appservice_port }}:{{ matrix_hookshot_appservice_port }}" - - "127.0.0.1:{{ matrix_hookshot_metrics_port }}:{{ matrix_hookshot_metrics_port }}" - - "127.0.0.1:{{ matrix_hookshot_webhook_port }}:{{ matrix_hookshot_webhook_port }}" - - "127.0.0.1:{{ matrix_hookshot_provisioning_port }}:{{ matrix_hookshot_provisioning_port }}" + - "{{ matrix_playbook_service_host_bind_interface_prefix }}{{ matrix_hookshot_appservice_port }}:{{ matrix_hookshot_appservice_port }}" + - "{{ matrix_playbook_service_host_bind_interface_prefix }}{{ matrix_hookshot_metrics_port }}:{{ matrix_hookshot_metrics_port }}" + - "{{ matrix_playbook_service_host_bind_interface_prefix }}{{ matrix_hookshot_webhook_port }}:{{ matrix_hookshot_webhook_port }}" + - "{{ matrix_playbook_service_host_bind_interface_prefix }}{{ matrix_hookshot_provisioning_port }}:{{ matrix_hookshot_provisioning_port }}" -matrix_hookshot_container_http_host_bind_ports: "{{ [] if matrix_nginx_proxy_enabled else matrix_hookshot_container_http_host_bind_ports_defaultmapping }}" +matrix_hookshot_container_http_host_bind_ports: "{{ matrix_hookshot_container_http_host_bind_ports_defaultmapping if matrix_playbook_service_host_bind_interface_prefix else [] }}" matrix_hookshot_provisioning_enabled: "{{ matrix_hookshot_provisioning_secret and matrix_dimension_enabled }}" @@ -1347,7 +1345,7 @@ matrix_mx_puppet_twitter_homeserver_token: "{{ '%s' | format(matrix_homeserver_g matrix_mx_puppet_twitter_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" -matrix_mx_puppet_twitter_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else ('127.0.0.1:' ~ matrix_mx_puppet_twitter_appservice_port) }}" +matrix_mx_puppet_twitter_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_mx_puppet_twitter_appservice_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # Postgres is the default, except if not using internal Postgres server matrix_mx_puppet_twitter_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -1607,7 +1605,7 @@ matrix_bot_maubot_registration_shared_secret: |- }[matrix_homeserver_implementation] }} -matrix_bot_maubot_management_interface_http_bind_port: "{{ '' if matrix_nginx_proxy_enabled else ('127.0.0.1:' + matrix_bot_maubot_management_interface_port | string) }}" +matrix_bot_maubot_management_interface_http_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_bot_maubot_management_interface_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # Postgres is the default, except if not using internal Postgres server matrix_bot_maubot_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -1700,8 +1698,12 @@ matrix_bot_postmoogle_container_image_self_build: "{{ matrix_architecture not in matrix_bot_postmoogle_ssl_path: |- {{ { - 'nginx-proxy': (matrix_ssl_config_dir_path if matrix_playbook_ssl_retrieval_method != 'none' else ''), - 'traefik': devture_traefik_certs_dumper_dumped_certificates_dir_path, + 'playbook-installed-traefik': devture_traefik_certs_dumper_dumped_certificates_dir_path, + 'other-traefik-container': devture_traefik_certs_dumper_dumped_certificates_dir_path, + 'playbook-installed-nginx': (matrix_ssl_config_dir_path if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'other-nginx-non-container': (matrix_ssl_config_dir_path if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'other-on-same-host': '', + 'other-on-another-host': '', 'none': '', }[matrix_playbook_reverse_proxy_type] }} @@ -1715,8 +1717,12 @@ matrix_playbook_bot_postmoogle_traefik_key: "{% for domain in matrix_bot_postmoo matrix_bot_postmoogle_tls_cert: |- {{ { - 'nginx-proxy': (matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert if matrix_playbook_ssl_retrieval_method != 'none' else ''), - 'traefik': matrix_playbook_bot_postmoogle_traefik_tls_cert, + 'playbook-installed-traefik': matrix_playbook_bot_postmoogle_traefik_tls_cert, + 'other-traefik-container': matrix_playbook_bot_postmoogle_traefik_tls_cert, + 'playbook-installed-nginx': (matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'other-nginx-non-container': (matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'other-on-same-host': '', + 'other-on-another-host': '', 'none': '', }[matrix_playbook_reverse_proxy_type] }} @@ -1724,8 +1730,12 @@ matrix_bot_postmoogle_tls_cert: |- matrix_bot_postmoogle_tls_key: |- {{ { - 'nginx-proxy': (matrix_playbook_bot_postmoogle_nginx_proxy_key if matrix_playbook_ssl_retrieval_method != 'none' else ''), - 'traefik': matrix_playbook_bot_postmoogle_traefik_key, + 'playbook-installed-traefik': matrix_playbook_bot_postmoogle_traefik_key, + 'other-traefik-container': matrix_playbook_bot_postmoogle_traefik_key, + 'playbook-installed-nginx': (matrix_playbook_bot_postmoogle_nginx_proxy_key if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'other-nginx-non-container': (matrix_playbook_bot_postmoogle_nginx_proxy_key if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'other-on-same-host': '', + 'other-on-another-host': '', 'none': '', }[matrix_playbook_reverse_proxy_type] }} @@ -1740,7 +1750,7 @@ matrix_bot_postmoogle_systemd_required_services_list: | + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + - (matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services | trim | split(' ') if matrix_playbook_reverse_proxy_type == 'traefik' and matrix_playbook_traefik_certs_dumper_role_enabled else []) + (matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services | trim | split(' ') if matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) }} # Postgres is the default, except if not using internal Postgres server @@ -1791,7 +1801,7 @@ matrix_bot_go_neb_systemd_required_services_list: | (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) }} -matrix_bot_go_neb_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:4050' }}" +matrix_bot_go_neb_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':4050') if matrix_playbook_service_host_bind_interface_prefix else '' }}" ###################################################################### # @@ -1938,8 +1948,8 @@ matrix_corporal_container_image_self_build: "{{ matrix_architecture not in ['amd # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-corporal over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-corporal's web-server ports to the local host. -matrix_corporal_container_http_gateway_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:41080' }}" -matrix_corporal_container_http_api_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:41081' }}" +matrix_corporal_container_http_gateway_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':41080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_corporal_container_http_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':41081') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_corporal_systemd_required_services_list: | {{ @@ -1982,8 +1992,12 @@ matrix_coturn_tls_enabled: "{{ matrix_playbook_ssl_retrieval_method != 'none' }} matrix_coturn_tls_cert_path: |- {{ { - 'nginx-proxy': '/fullchain.pem', - 'traefik': '/certificate.crt', + 'playbook-installed-traefik': '/certificate.crt', + 'other-traefik-container': '/certificate.crt', + 'playbook-installed-nginx': '/fullchain.pem', + 'other-nginx-non-container': '/fullchain.pem', + 'other-on-same-host': '', + 'other-on-another-host': '', 'none': '', }[matrix_playbook_reverse_proxy_type] }} @@ -1991,8 +2005,12 @@ matrix_coturn_tls_cert_path: |- matrix_coturn_tls_key_path: |- {{ { - 'nginx-proxy': '/privkey.pem', - 'traefik': '/privatekey.key', + 'playbook-installed-traefik': '/privatekey.key', + 'other-traefik-container': '/privatekey.key', + 'playbook-installed-nginx': '/privkey.pem', + 'other-nginx-non-container': '/privkey.pem', + 'other-on-same-host': '', + 'other-on-another-host': '', 'none': '', }[matrix_playbook_reverse_proxy_type] }} @@ -2011,7 +2029,7 @@ matrix_coturn_container_additional_volumes: | 'dst': '/privkey.pem', 'options': 'ro', }, - ] if matrix_playbook_reverse_proxy_type == 'nginx-proxy' else [] + ] if matrix_playbook_reverse_proxy_type in ['playbook-installed-nginx', 'other-nginx-non-container'] else [] ) + ( @@ -2026,7 +2044,7 @@ matrix_coturn_container_additional_volumes: | 'dst': '/privatekey.key', 'options': 'ro', }, - ] if matrix_playbook_reverse_proxy_type == 'traefik' and matrix_playbook_traefik_certs_dumper_role_enabled else [] + ] if matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else [] ) }} @@ -2034,7 +2052,7 @@ matrix_coturn_systemd_required_services_list: | {{ ['docker.service'] + - ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type == 'traefik' and matrix_playbook_traefik_certs_dumper_role_enabled else []) + ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) }} ###################################################################### @@ -2056,7 +2074,7 @@ matrix_dimension_container_image_self_build: "{{ matrix_architecture != 'amd64' # Normally, matrix-nginx-proxy is enabled and nginx can reach Dimension over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the Dimension HTTP port to the local host. -matrix_dimension_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8184' }}" +matrix_dimension_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8184') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_dimension_homeserver_federationUrl: "{{ matrix_homeserver_container_federation_url }}" @@ -2090,7 +2108,7 @@ matrix_dimension_database_password: "{{ '%s' | format(matrix_homeserver_generic_ matrix_etherpad_enabled: false -matrix_etherpad_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9001' }}" +matrix_etherpad_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9001') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_etherpad_base_url: "{{ 'https://' + matrix_server_fqn_dimension + matrix_etherpad_public_endpoint if matrix_etherpad_mode == 'dimension' else 'https://' + matrix_server_fqn_etherpad + '/' }}" @@ -2151,11 +2169,11 @@ matrix_jitsi_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach jitsi/web over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the Jitsi HTTP port to the local host. -matrix_jitsi_web_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:13080' }}" +matrix_jitsi_web_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':13080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" -matrix_jitsi_jvb_container_colibri_ws_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:13090' }}" +matrix_jitsi_jvb_container_colibri_ws_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':13090') if matrix_playbook_service_host_bind_interface_prefix else '' }}" -matrix_jitsi_prosody_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:5280' }}" +matrix_jitsi_prosody_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':5280') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_jitsi_jibri_xmpp_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'jibri', rounds=655555) | to_uuid }}" matrix_jitsi_jicofo_auth_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'jicofo', rounds=655555) | to_uuid }}" @@ -2245,7 +2263,7 @@ matrix_ma1sd_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" # Normally, matrix-nginx-proxy is enabled and nginx can reach ma1sd over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # ma1sd's web-server port. -matrix_ma1sd_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:' + matrix_ma1sd_container_port | string }}" +matrix_ma1sd_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_ma1sd_container_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # We enable Synapse integration via its Postgres database by default. @@ -2259,7 +2277,7 @@ matrix_ma1sd_dns_overwrite_enabled: true matrix_ma1sd_dns_overwrite_homeserver_client_name: "{{ matrix_server_fqn_matrix }}" # The `matrix_ma1sd_dns_overwrite_homeserver_client_value` value when matrix_nginx_proxy_enabled is false covers the general case, # but may be inaccurate if matrix-corporal is enabled. -matrix_ma1sd_dns_overwrite_homeserver_client_value: "{{ ('http://' + matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container) if matrix_nginx_proxy_enabled else matrix_homeserver_container_url }}" +matrix_ma1sd_dns_overwrite_homeserver_client_value: "{{ matrix_homeserver_container_url }}" # By default, we send mail through the `matrix-mailer` service. matrix_ma1sd_threepid_medium_email_identity_from: "{{ matrix_mailer_sender_address }}" @@ -2301,10 +2319,24 @@ matrix_ma1sd_database_password: "{{ '%s' | format(matrix_homeserver_generic_secr # ###################################################################### -# By default, this playbook sets up a reverse-proxy nginx proxy server on TCP ports 80, 443 and 8448. -# This is fine if you're dedicating the whole server to Matrix. -# If that's not the case, you may wish to disable this and take care of proxying yourself. -matrix_nginx_proxy_enabled: true +# This playbook installs its own nginx if +# - it's explicitly enabled +# - Traefik is in use. Not all services are Traefik-native yet, so we use reverse-proxy to some via a local-only matrix-nginx-proxy +matrix_nginx_proxy_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-installed-nginx', 'playbook-installed-traefik', 'other-traefik-container'] }}" + +# matrix-nginx-proxy is only to handle HTTPS only if it's the chosen reverse-proxy. +# It may be enabled even if it's not chosen. See `matrix_nginx_proxy_enabled`. +matrix_ssl_retrieval_method: "{{ 'lets-encrypt' if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else 'none' }}" +matrix_nginx_proxy_https_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' }}" + +# matrix-nginx-proxy is to publish ports only if it's the chosen reverse-proxy. +# It may be enabled even if it's not chosen. See `matrix_nginx_proxy_enabled`. +matrix_nginx_proxy_container_http_host_bind_port: "{{ '80' if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else '' }}" +matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else '' }}" + +# matrix-nginx-proxy is to trust reverse-proxy forwarded protocol and headers, unless it's the "main" (chosen) reverse-proxy +matrix_nginx_proxy_trust_forwarded_proto: "{{ matrix_playbook_reverse_proxy_type != 'playbook-installed-nginx' }}" +matrix_nginx_proxy_x_forwarded_for: "{{ '$remote_addr' if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else '$proxy_add_x_forwarded_for' }}" matrix_nginx_proxy_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" @@ -2399,8 +2431,6 @@ matrix_nginx_proxy_proxy_conduit_federation_api_addr_sans_container: "127.0.0.1: # When matrix-nginx-proxy is disabled, the actual port number that the vhost uses may begin to matter. matrix_nginx_proxy_proxy_matrix_federation_port: "{{ matrix_federation_public_port }}" -matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port }}" - matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled: "{{ matrix_ma1sd_enabled }}" matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container }}" matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container }}" @@ -2828,7 +2858,7 @@ matrix_sygnal_enabled: false # If someone instals Prometheus via the playbook, they most likely wish to monitor Sygnal. matrix_sygnal_metrics_prometheus_enabled: "{{ matrix_prometheus_enabled }}" -matrix_sygnal_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:6000' }}" +matrix_sygnal_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':6000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" ###################################################################### # @@ -2844,7 +2874,7 @@ matrix_sygnal_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enable matrix_ntfy_enabled: false -matrix_ntfy_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:2586' }}" +matrix_ntfy_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':2586') if matrix_playbook_service_host_bind_interface_prefix else '' }}" ###################################################################### # @@ -2881,7 +2911,7 @@ matrix_client_element_container_image_self_build: "{{ matrix_architecture not in # Normally, matrix-nginx-proxy is enabled and nginx can reach Element over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the Element HTTP port to the local host. -matrix_client_element_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8765' }}" +matrix_client_element_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8765') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_client_element_default_hs_url: "{{ matrix_homeserver_url }}" matrix_client_element_default_is_url: "{{ matrix_identity_server_url }}" @@ -2926,7 +2956,7 @@ matrix_client_hydrogen_container_image_self_build: "{{ matrix_architecture not i # Normally, matrix-nginx-proxy is enabled and nginx can reach Hydrogen over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the HTTP port to the local host. -matrix_client_hydrogen_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8768' }}" +matrix_client_hydrogen_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8768') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_client_hydrogen_default_hs_url: "{{ matrix_homeserver_url }}" @@ -2951,7 +2981,7 @@ matrix_client_cinny_container_image_self_build: "{{ matrix_architecture not in [ # Normally, matrix-nginx-proxy is enabled and nginx can reach Cinny over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the HTTP port to the local host. -matrix_client_cinny_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8080' }}" +matrix_client_cinny_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_client_cinny_default_hs_url: "{{ matrix_homeserver_url }}" @@ -2980,13 +3010,13 @@ matrix_synapse_account_threepid_delegates_msisdn: "{{ 'http://matrix-ma1sd:' + m matrix_synapse_container_federation_api_tls_host_bind_port: "{{ matrix_federation_public_port if (matrix_synapse_federation_enabled and matrix_synapse_tls_federation_listener_enabled) else '' }}" # # For exposing the Synapse Metrics API's port (plain HTTP) to the local host. -matrix_synapse_container_metrics_api_host_bind_port: "{{ '127.0.0.1:9100' if (matrix_synapse_metrics_enabled and not matrix_nginx_proxy_enabled) else '' }}" +matrix_synapse_container_metrics_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9100') if matrix_synapse_metrics_enabled and matrix_playbook_service_host_bind_interface_prefix else '' }}" # # For exposing the Synapse Manhole port (plain HTTP) to the local host. -matrix_synapse_container_manhole_api_host_bind_port: "{{ '127.0.0.1:9000' if matrix_synapse_manhole_enabled else '' }}" +matrix_synapse_container_manhole_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9000') if matrix_synapse_metrics_enabled and matrix_playbook_service_host_bind_interface_prefix else '' }}" # # For exposing the Synapse worker (and metrics) ports to the local host. -matrix_synapse_workers_container_host_bind_address: "{{ '127.0.0.1' if (matrix_synapse_workers_enabled and not matrix_nginx_proxy_enabled) else '' }}" +matrix_synapse_workers_container_host_bind_address: "{{ matrix_playbook_service_host_bind_interface_prefix[0:-1] if (matrix_synapse_workers_enabled and matrix_playbook_service_host_bind_interface_prefix) else '' }}" matrix_synapse_database_host: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" matrix_synapse_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'synapse.db', rounds=655555) | to_uuid }}" @@ -3075,8 +3105,8 @@ matrix_synapse_reverse_proxy_companion_enabled: "{{ matrix_synapse_enabled }}" matrix_synapse_reverse_proxy_companion_client_api_client_max_body_size_mb: "{{ matrix_synapse_max_upload_size_mb }}" -matrix_synapse_reverse_proxy_companion_container_client_api_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8008' }}" -matrix_synapse_reverse_proxy_companion_container_federation_api_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8048' }}" +matrix_synapse_reverse_proxy_companion_container_client_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8008') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_synapse_reverse_proxy_companion_container_federation_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8048') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_synapse_reverse_proxy_companion_synapse_workers_enabled: "{{ matrix_synapse_workers_enabled }}" matrix_synapse_reverse_proxy_companion_synapse_workers_list: "{{ matrix_synapse_workers_enabled_list }}" @@ -3110,7 +3140,7 @@ matrix_synapse_admin_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach Synapse Admin over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Synapse Admin's HTTP port to the local host. -matrix_synapse_admin_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8766' }}" +matrix_synapse_admin_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8766') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" @@ -3219,7 +3249,7 @@ matrix_prometheus_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach Prometheus over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Prometheus' HTTP port to the local host. -matrix_prometheus_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9090' }}" +matrix_prometheus_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':99090005') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_prometheus_scraper_synapse_enabled: "{{ matrix_synapse_enabled and matrix_synapse_metrics_enabled }}" matrix_prometheus_scraper_synapse_targets: ['matrix-synapse:{{ matrix_synapse_metrics_port }}'] @@ -3256,7 +3286,7 @@ matrix_grafana_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach Grafana over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Grafana's HTTP port to the local host. -matrix_grafana_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:3000' }}" +matrix_grafana_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':3000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_grafana_dashboard_download_urls: | {{ @@ -3295,7 +3325,7 @@ matrix_registration_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-registration over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-registration's HTTP port to the local host. -matrix_registration_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8767' }}" +matrix_registration_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8767') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_registration_riot_instance: "{{ ('https://' + matrix_server_fqn_element) if matrix_client_element_enabled else 'https://riot.im/app/' }}" @@ -3346,12 +3376,10 @@ matrix_dendrite_enabled: "{{ matrix_homeserver_implementation == 'dendrite' }}" # you can expose Dendrite's ports to the host. # # For exposing Dendrite's plain HTTP server to the local host. -matrix_dendrite_container_http_host_bind_address: "{{ '' if matrix_nginx_proxy_enabled else ('127.0.0.1:' + matrix_dendrite_http_bind_port | string) }}" +matrix_dendrite_container_http_host_bind_address: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_dendrite_http_bind_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # # For exposing Dendrite's HTTPS server to the local host. -matrix_dendrite_container_https_host_bind_address: "{{ '' if matrix_nginx_proxy_enabled or not matrix_dendrite_https_bind_port else ('127.0.0.1:' + matrix_dendrite_https_bind_port | string) }}" - -matrix_dendrite_sync_api_real_ip_header: "{{ 'X-Forwarded-For' if matrix_nginx_proxy_enabled else '' }}" +matrix_dendrite_container_https_host_bind_address: "{{ '' if not matrix_dendrite_https_bind_port or not matrix_playbook_service_host_bind_interface_prefix else (matrix_playbook_service_host_bind_interface_prefix + matrix_dendrite_https_bind_port | string) }}" matrix_dendrite_client_api_registration_shared_secret: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'dendrite.rss', rounds=655555) | to_uuid }}" @@ -3486,9 +3514,9 @@ matrix_user_verification_service_systemd_required_services_list: | # If Jitsi is managed by this playbook we can use the docker network - no need to expose a port. # If Jitsi is not managed by this playbook, or you otherwise have a need for it, you can expose -# matrix-user-verfification-services's client-server port to the local host port 3003. +# matrix-user-verfification-services's client-server port to port 3003. # By default Matrix-User-Verification-Service binds to port 3000, which collides with grafana, therefore this uses port 3003. -matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') else '127.0.0.1:3003' }}" +matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') else matrix_playbook_service_host_bind_interface_prefix ~ ':3003' }}" # URL exposed in the docker network matrix_user_verification_service_container_url: "http://{{ matrix_user_verification_service_container_name }}:3000" @@ -3513,7 +3541,7 @@ matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homese # To completely disable the Traefik role from running, use `matrix_playbook_traefik_role_enabled: false`. # See the comment there for more details about why we have both `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`. -devture_traefik_enabled: "{{ matrix_playbook_traefik_role_enabled }}" +devture_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-installed-traefik' }}" devture_traefik_uid: "{{ matrix_user_uid }}" devture_traefik_gid: "{{ matrix_user_gid }}" diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 025e7363..e74fa4f1 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -265,6 +265,48 @@ matrix_well_known_matrix_support_enabled: false matrix_homeserver_container_extra_arguments_auto: [] matrix_homeserver_app_service_config_files_auto: [] +# Specifies the type of reverse-proxy used by the playbook. +# +# Changing this has an effect on whether a reverse-proxy is installed at all and what its type is, +# as well as how all other services are configured. +# +# Valid options and a description of their behavior: +# +# - `playbook-installed-traefik` +# - the playbook will install devture-traefik for SSL termination +# - it will also install matrix-nginx-proxy in local-only mode, while we migrate the rest of the services to a Traefik-native mode of working +# +# - `playbook-installed-nginx` +# - the playbook will install matrix-nginx-proxy and do SSL termination with Certbot +# +# - `other-traefik-container` +# - Traefik will be used, but it's not installed by this playbook. +# - you should make sure it's compatible with what the playbook would have configured (web, web-secure, matrix-federation entrypoints, etc.) +# - you may wish to set `matrix_playbook_reverse_proxyable_services_additional_network` to the name of your Traefik network +# - you may wish to enable `devture_traefik_certs_dumper_enabled` and point it to your Traefik's SSL certificates (`devture_traefik_certs_dumper_ssl_dir_path`) +# +# - `other-nginx-non-container` +# - the playbook will not install matrix-nginx-proxy +# - however, it will still dump some nginx configuration in /matrix/nginx/conf.d +# - these configs are meant to be included into a locally-installed (without a container) nginx server +# - all container services are exposed locally (e.g. `-p 127.0.0.1:8080:8080`) +# +# - `other-on-same-host` +# - like other-nginx-non-container, but supposedly won't generate useless configuration in /matrix/nginx/conf.d in the future +# +# - `other-on-another-host` +# - like other-on-same-host, but services are exposed on all interfaces (e.g. `-p 0.0.0.0:8080:8080`) +# - configurable via `matrix_playbook_service_host_bind_interface_prefix` +# +# - `none` +# - no reverse-proxy will be installed +# - no nginx configuration will be dumped in /matrix/nginx/conf.d +# - no port exposure will be done for any of the container services +# - it's up to you to expose the ports you want, etc. +matrix_playbook_reverse_proxy_type: playbook-installed-nginx + +matrix_playbook_service_host_bind_interface_prefix: "{{ '' if matrix_playbook_reverse_proxy_type not in ['other-nginx-non-container', 'other-on-same-host', 'other-on-another-host'] else ('0.0.0.0:' if matrix_playbook_reverse_proxy_type == 'other-on-another-host' else '127.0.0.1:') }}" + # Variables to Control which parts of our roles run. run_postgres_import: true run_postgres_upgrade: true diff --git a/roles/custom/matrix-base/tasks/validate_config.yml b/roles/custom/matrix-base/tasks/validate_config.yml index b3551e2e..5f63c1c6 100644 --- a/roles/custom/matrix-base/tasks/validate_config.yml +++ b/roles/custom/matrix-base/tasks/validate_config.yml @@ -51,6 +51,11 @@ msg: "Detected that variable matrix_architecture {{ matrix_architecture }} appears to be set incorrectly. See docs/alternative-architectures.md. Server appears to be {{ ansible_architecture }}." when: matrix_architecture not in ['amd64', 'arm32', 'arm64'] +- name: Fail if matrix_playbook_reverse_proxy_type is set incorrectly + ansible.builtin.fail: + msg: "Detected that variable matrix_playbook_reverse_proxy_type (current value: `{{ matrix_playbook_reverse_proxy_type }}`) appears to be set incorrectly. See roles/custom/matrix-base/defaults/main.yml for valid choices." + when: matrix_playbook_reverse_proxy_type not in ['playbook-installed-traefik', 'playbook-installed-nginx', 'other-traefik-container', 'other-nginx-non-container', 'other-on-same-host', 'other-on-another-host', 'none'] + - name: Fail if uppercase domain used ansible.builtin.fail: msg: "Detected that you're using an uppercase domain name - `{{ item }}`. This will cause trouble. Please use all-lowercase!" diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 6a495f8f..5e6d7d3b 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -84,7 +84,7 @@ matrix_dendrite_max_file_size_bytes: 10485760 # Controls which HTTP header (e.g. 'X-Forwarded-For', 'X-Real-IP') to inspect to find the real remote IP address of the client. # This is likely required if Dendrite is running behind a reverse proxy server. -matrix_dendrite_sync_api_real_ip_header: '' +matrix_dendrite_sync_api_real_ip_header: 'X-Forwarded-For' # The tmpfs at /tmp needs to be large enough to handle multiple concurrent file uploads. matrix_dendrite_tmp_directory_size_mb: 500 From c773a005ae3253fc9e22f5bd976e47929addaecc Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 16:08:55 +0200 Subject: [PATCH 106/382] Fix broken link to Matrix User Verification Service --- docs/configuring-playbook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index 3e904abc..1f3cb7a4 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -99,7 +99,7 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up Matrix Corporal](configuring-playbook-matrix-corporal.md) (optional, advanced) -- [Matrix User Verification Service](docs/configuring-playbook-user-verification-service.md) (optional, advanced) +- [Matrix User Verification Service](configuring-playbook-user-verification-service.md) (optional, advanced) ### Bridging other networks From a1ef28681ac6e388a869fae6dd43c6262006ad71 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 10 Feb 2023 14:40:50 +0000 Subject: [PATCH 107/382] Update Hydrogen 0.3.6 -> 0.3.7 --- roles/custom/matrix-client-hydrogen/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-hydrogen/defaults/main.yml b/roles/custom/matrix-client-hydrogen/defaults/main.yml index c3f2ab95..6ee304cf 100644 --- a/roles/custom/matrix-client-hydrogen/defaults/main.yml +++ b/roles/custom/matrix-client-hydrogen/defaults/main.yml @@ -6,7 +6,7 @@ matrix_client_hydrogen_enabled: true matrix_client_hydrogen_container_image_self_build: false matrix_client_hydrogen_container_image_self_build_repo: "https://github.com/vector-im/hydrogen-web.git" -matrix_client_hydrogen_version: v0.3.6 +matrix_client_hydrogen_version: v0.3.7 matrix_client_hydrogen_docker_image: "{{ matrix_client_hydrogen_docker_image_name_prefix }}vector-im/hydrogen-web:{{ matrix_client_hydrogen_version }}" matrix_client_hydrogen_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_hydrogen_container_image_self_build else 'ghcr.io/' }}" matrix_client_hydrogen_docker_image_force_pull: "{{ matrix_client_hydrogen_docker_image.endswith(':latest') }}" From 97f65e8dffcf979a296cc99dfecaf744392a0116 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 19:36:06 +0200 Subject: [PATCH 108/382] Minor fixes to allow for Traefik without SSL --- group_vars/matrix_servers | 2 +- roles/custom/matrix-base/defaults/main.yml | 13 ++++++++----- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 62771b6c..2bcaa73c 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -38,7 +38,7 @@ matrix_playbook_traefik_role_enabled: true # installed in another way. matrix_playbook_traefik_labels_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] }}" -matrix_playbook_traefik_certs_dumper_role_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] }}" +matrix_playbook_traefik_certs_dumper_role_enabled: "{{ (matrix_playbook_reverse_proxy_type == 'playbook-installed-traefik' and devture_traefik_config_entrypoint_web_secure_enabled) or matrix_playbook_reverse_proxy_type == 'other-traefik-container' }}" # Controls the additional network that reverse-proxyable services will be connected to. matrix_playbook_reverse_proxyable_services_additional_network: "{{ devture_traefik_container_network if devture_traefik_enabled else '' }}" diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index e74fa4f1..7e76a9e7 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -273,16 +273,19 @@ matrix_homeserver_app_service_config_files_auto: [] # Valid options and a description of their behavior: # # - `playbook-installed-traefik` -# - the playbook will install devture-traefik for SSL termination +# - the playbook will install devture-traefik +# - Traefik will do SSL termination, unless you disable it (e.g. `devture_traefik_config_entrypoint_web_secure_enabled: false`) # - it will also install matrix-nginx-proxy in local-only mode, while we migrate the rest of the services to a Traefik-native mode of working # # - `playbook-installed-nginx` -# - the playbook will install matrix-nginx-proxy and do SSL termination with Certbot +# - the playbook will install matrix-nginx-proxy +# - matrix-nginx-proxy will do SSL termination with Certbot, unless you change that (see `matrix_ssl_retrieval_method`) # # - `other-traefik-container` -# - Traefik will be used, but it's not installed by this playbook. -# - you should make sure it's compatible with what the playbook would have configured (web, web-secure, matrix-federation entrypoints, etc.) -# - you may wish to set `matrix_playbook_reverse_proxyable_services_additional_network` to the name of your Traefik network +# - this playbook will not install Traefik +# - nevertheless, the playbook expects that you would install Traefik yourself via other means +# - you should make sure your Traefik configuration is compatible with what the playbook would have configured (web, web-secure, matrix-federation entrypoints, etc.) +# - you need to set `matrix_playbook_reverse_proxyable_services_additional_network` to the name of your Traefik network # - you may wish to enable `devture_traefik_certs_dumper_enabled` and point it to your Traefik's SSL certificates (`devture_traefik_certs_dumper_ssl_dir_path`) # # - `other-nginx-non-container` From 7142ff422dfe8aad313f77f9fd19a51d50799a4f Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 19:39:15 +0200 Subject: [PATCH 109/382] Ensure matrix_user_verification_service_uvs_access_token is always defined The playbook tries to avoid such variables which are sometimes defined and sometimes not. We'd rather not check for `is defined`. --- roles/custom/matrix-user-verification-service/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-user-verification-service/defaults/main.yml b/roles/custom/matrix-user-verification-service/defaults/main.yml index 52b30a28..5b3611b8 100644 --- a/roles/custom/matrix-user-verification-service/defaults/main.yml +++ b/roles/custom/matrix-user-verification-service/defaults/main.yml @@ -32,7 +32,7 @@ matrix_user_verification_service_systemd_service_name: "{{ matrix_user_verificat ## REQUIRED # Homeserver client API admin token (synapse only)- Required for the service to verify room membership -# matrix_user_verification_service_uvs_access_token: +matrix_user_verification_service_uvs_access_token: '' # homeserver client api url matrix_user_verification_service_uvs_homeserver_url: "{{ matrix_homeserver_container_url }}" From ad22bdb884e6cda2308412aca5f72e0e53d04216 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 10 Feb 2023 19:39:55 +0200 Subject: [PATCH 110/382] Do not run matrix-user-verification-service validation tasks unless the service is enabled --- roles/custom/matrix-user-verification-service/tasks/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-user-verification-service/tasks/main.yml b/roles/custom/matrix-user-verification-service/tasks/main.yml index 92686036..6eaaaf23 100644 --- a/roles/custom/matrix-user-verification-service/tasks/main.yml +++ b/roles/custom/matrix-user-verification-service/tasks/main.yml @@ -1,7 +1,7 @@ --- - block: - - when: matrix_jitsi_enabled | bool + - when: run_setup | bool and matrix_user_verification_service_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: run_setup | bool and matrix_user_verification_service_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" From 8309a2130354d407171c489b803d24175c446b2b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 11 Feb 2023 08:44:11 +0200 Subject: [PATCH 111/382] Rename reverse proxy types and fix Hookshot http/https urlPrefix issue --- group_vars/matrix_servers | 52 ++++++++++--------- roles/custom/matrix-base/defaults/main.yml | 6 +-- .../matrix-base/tasks/validate_config.yml | 2 +- 3 files changed, 31 insertions(+), 29 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 2bcaa73c..5a79ccdc 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -36,15 +36,17 @@ matrix_playbook_traefik_role_enabled: true # This is separate from `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`, # because you may wish to disable Traefik installation by the playbook, yet still use Traefik # installed in another way. -matrix_playbook_traefik_labels_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] }}" +matrix_playbook_traefik_labels_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" -matrix_playbook_traefik_certs_dumper_role_enabled: "{{ (matrix_playbook_reverse_proxy_type == 'playbook-installed-traefik' and devture_traefik_config_entrypoint_web_secure_enabled) or matrix_playbook_reverse_proxy_type == 'other-traefik-container' }}" +matrix_playbook_traefik_certs_dumper_role_enabled: "{{ (matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' and devture_traefik_config_entrypoint_web_secure_enabled) or matrix_playbook_reverse_proxy_type == 'other-traefik-container' }}" # Controls the additional network that reverse-proxyable services will be connected to. matrix_playbook_reverse_proxyable_services_additional_network: "{{ devture_traefik_container_network if devture_traefik_enabled else '' }}" matrix_playbook_ssl_retrieval_method: "{{ 'lets-encrypt' if matrix_playbook_traefik_certs_dumper_role_enabled else matrix_ssl_retrieval_method }}" +matrix_playbook_ssl_enabled: "{{ matrix_playbook_ssl_retrieval_method in ['lets-encrypt', 'self-signed', 'manually-managed'] }}" + ######################################################################## # # # /Playbook # @@ -1269,7 +1271,7 @@ matrix_hookshot_metrics_enabled: "{{ matrix_prometheus_enabled }}" matrix_hookshot_urlprefix_port_enabled: "{{ matrix_nginx_proxy_container_https_host_bind_port == 443 if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_https_host_bind_port == 80 }}" matrix_hookshot_urlprefix_port: ":{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}" -matrix_hookshot_urlprefix: "http{{ 's' if matrix_nginx_proxy_https_enabled else '' }}://{{ matrix_server_fqn_matrix }}{{ matrix_hookshot_urlprefix_port if matrix_hookshot_urlprefix_port_enabled else '' }}" +matrix_hookshot_urlprefix: "{{ 'https' if matrix_playbook_ssl_enabled else 'http' }}://{{ matrix_server_fqn_matrix }}{{ matrix_hookshot_urlprefix_port if matrix_hookshot_urlprefix_port_enabled else '' }}" ###################################################################### # @@ -1698,9 +1700,9 @@ matrix_bot_postmoogle_container_image_self_build: "{{ matrix_architecture not in matrix_bot_postmoogle_ssl_path: |- {{ { - 'playbook-installed-traefik': devture_traefik_certs_dumper_dumped_certificates_dir_path, + 'playbook-managed-traefik': devture_traefik_certs_dumper_dumped_certificates_dir_path, 'other-traefik-container': devture_traefik_certs_dumper_dumped_certificates_dir_path, - 'playbook-installed-nginx': (matrix_ssl_config_dir_path if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'playbook-managed-nginx': (matrix_ssl_config_dir_path if matrix_playbook_ssl_retrieval_method != 'none' else ''), 'other-nginx-non-container': (matrix_ssl_config_dir_path if matrix_playbook_ssl_retrieval_method != 'none' else ''), 'other-on-same-host': '', 'other-on-another-host': '', @@ -1717,9 +1719,9 @@ matrix_playbook_bot_postmoogle_traefik_key: "{% for domain in matrix_bot_postmoo matrix_bot_postmoogle_tls_cert: |- {{ { - 'playbook-installed-traefik': matrix_playbook_bot_postmoogle_traefik_tls_cert, + 'playbook-managed-traefik': matrix_playbook_bot_postmoogle_traefik_tls_cert, 'other-traefik-container': matrix_playbook_bot_postmoogle_traefik_tls_cert, - 'playbook-installed-nginx': (matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'playbook-managed-nginx': (matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert if matrix_playbook_ssl_retrieval_method != 'none' else ''), 'other-nginx-non-container': (matrix_playbook_bot_postmoogle_nginx_proxy_tls_cert if matrix_playbook_ssl_retrieval_method != 'none' else ''), 'other-on-same-host': '', 'other-on-another-host': '', @@ -1730,9 +1732,9 @@ matrix_bot_postmoogle_tls_cert: |- matrix_bot_postmoogle_tls_key: |- {{ { - 'playbook-installed-traefik': matrix_playbook_bot_postmoogle_traefik_key, + 'playbook-managed-traefik': matrix_playbook_bot_postmoogle_traefik_key, 'other-traefik-container': matrix_playbook_bot_postmoogle_traefik_key, - 'playbook-installed-nginx': (matrix_playbook_bot_postmoogle_nginx_proxy_key if matrix_playbook_ssl_retrieval_method != 'none' else ''), + 'playbook-managed-nginx': (matrix_playbook_bot_postmoogle_nginx_proxy_key if matrix_playbook_ssl_retrieval_method != 'none' else ''), 'other-nginx-non-container': (matrix_playbook_bot_postmoogle_nginx_proxy_key if matrix_playbook_ssl_retrieval_method != 'none' else ''), 'other-on-same-host': '', 'other-on-another-host': '', @@ -1750,7 +1752,7 @@ matrix_bot_postmoogle_systemd_required_services_list: | + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + - (matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services | trim | split(' ') if matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) + (matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services | trim | split(' ') if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) }} # Postgres is the default, except if not using internal Postgres server @@ -1992,9 +1994,9 @@ matrix_coturn_tls_enabled: "{{ matrix_playbook_ssl_retrieval_method != 'none' }} matrix_coturn_tls_cert_path: |- {{ { - 'playbook-installed-traefik': '/certificate.crt', + 'playbook-managed-traefik': '/certificate.crt', 'other-traefik-container': '/certificate.crt', - 'playbook-installed-nginx': '/fullchain.pem', + 'playbook-managed-nginx': '/fullchain.pem', 'other-nginx-non-container': '/fullchain.pem', 'other-on-same-host': '', 'other-on-another-host': '', @@ -2005,9 +2007,9 @@ matrix_coturn_tls_cert_path: |- matrix_coturn_tls_key_path: |- {{ { - 'playbook-installed-traefik': '/privatekey.key', + 'playbook-managed-traefik': '/privatekey.key', 'other-traefik-container': '/privatekey.key', - 'playbook-installed-nginx': '/privkey.pem', + 'playbook-managed-nginx': '/privkey.pem', 'other-nginx-non-container': '/privkey.pem', 'other-on-same-host': '', 'other-on-another-host': '', @@ -2029,7 +2031,7 @@ matrix_coturn_container_additional_volumes: | 'dst': '/privkey.pem', 'options': 'ro', }, - ] if matrix_playbook_reverse_proxy_type in ['playbook-installed-nginx', 'other-nginx-non-container'] else [] + ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] else [] ) + ( @@ -2044,7 +2046,7 @@ matrix_coturn_container_additional_volumes: | 'dst': '/privatekey.key', 'options': 'ro', }, - ] if matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else [] + ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else [] ) }} @@ -2052,7 +2054,7 @@ matrix_coturn_systemd_required_services_list: | {{ ['docker.service'] + - ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type in ['playbook-installed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) + ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) }} ###################################################################### @@ -2322,21 +2324,21 @@ matrix_ma1sd_database_password: "{{ '%s' | format(matrix_homeserver_generic_secr # This playbook installs its own nginx if # - it's explicitly enabled # - Traefik is in use. Not all services are Traefik-native yet, so we use reverse-proxy to some via a local-only matrix-nginx-proxy -matrix_nginx_proxy_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-installed-nginx', 'playbook-installed-traefik', 'other-traefik-container'] }}" +matrix_nginx_proxy_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'playbook-managed-traefik', 'other-traefik-container'] }}" # matrix-nginx-proxy is only to handle HTTPS only if it's the chosen reverse-proxy. # It may be enabled even if it's not chosen. See `matrix_nginx_proxy_enabled`. -matrix_ssl_retrieval_method: "{{ 'lets-encrypt' if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else 'none' }}" -matrix_nginx_proxy_https_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' }}" +matrix_ssl_retrieval_method: "{{ 'lets-encrypt' if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'none' }}" +matrix_nginx_proxy_https_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' }}" # matrix-nginx-proxy is to publish ports only if it's the chosen reverse-proxy. # It may be enabled even if it's not chosen. See `matrix_nginx_proxy_enabled`. -matrix_nginx_proxy_container_http_host_bind_port: "{{ '80' if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else '' }}" -matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else '' }}" +matrix_nginx_proxy_container_http_host_bind_port: "{{ '80' if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else '' }}" +matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else '' }}" # matrix-nginx-proxy is to trust reverse-proxy forwarded protocol and headers, unless it's the "main" (chosen) reverse-proxy -matrix_nginx_proxy_trust_forwarded_proto: "{{ matrix_playbook_reverse_proxy_type != 'playbook-installed-nginx' }}" -matrix_nginx_proxy_x_forwarded_for: "{{ '$remote_addr' if matrix_playbook_reverse_proxy_type == 'playbook-installed-nginx' else '$proxy_add_x_forwarded_for' }}" +matrix_nginx_proxy_trust_forwarded_proto: "{{ matrix_playbook_reverse_proxy_type != 'playbook-managed-nginx' }}" +matrix_nginx_proxy_x_forwarded_for: "{{ '$remote_addr' if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else '$proxy_add_x_forwarded_for' }}" matrix_nginx_proxy_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" @@ -3541,7 +3543,7 @@ matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homese # To completely disable the Traefik role from running, use `matrix_playbook_traefik_role_enabled: false`. # See the comment there for more details about why we have both `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`. -devture_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-installed-traefik' }}" +devture_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' }}" devture_traefik_uid: "{{ matrix_user_uid }}" devture_traefik_gid: "{{ matrix_user_gid }}" diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 7e76a9e7..8cf63b9b 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -272,12 +272,12 @@ matrix_homeserver_app_service_config_files_auto: [] # # Valid options and a description of their behavior: # -# - `playbook-installed-traefik` +# - `playbook-managed-traefik` # - the playbook will install devture-traefik # - Traefik will do SSL termination, unless you disable it (e.g. `devture_traefik_config_entrypoint_web_secure_enabled: false`) # - it will also install matrix-nginx-proxy in local-only mode, while we migrate the rest of the services to a Traefik-native mode of working # -# - `playbook-installed-nginx` +# - `playbook-managed-nginx` # - the playbook will install matrix-nginx-proxy # - matrix-nginx-proxy will do SSL termination with Certbot, unless you change that (see `matrix_ssl_retrieval_method`) # @@ -306,7 +306,7 @@ matrix_homeserver_app_service_config_files_auto: [] # - no nginx configuration will be dumped in /matrix/nginx/conf.d # - no port exposure will be done for any of the container services # - it's up to you to expose the ports you want, etc. -matrix_playbook_reverse_proxy_type: playbook-installed-nginx +matrix_playbook_reverse_proxy_type: playbook-managed-nginx matrix_playbook_service_host_bind_interface_prefix: "{{ '' if matrix_playbook_reverse_proxy_type not in ['other-nginx-non-container', 'other-on-same-host', 'other-on-another-host'] else ('0.0.0.0:' if matrix_playbook_reverse_proxy_type == 'other-on-another-host' else '127.0.0.1:') }}" diff --git a/roles/custom/matrix-base/tasks/validate_config.yml b/roles/custom/matrix-base/tasks/validate_config.yml index 5f63c1c6..cd6d20cd 100644 --- a/roles/custom/matrix-base/tasks/validate_config.yml +++ b/roles/custom/matrix-base/tasks/validate_config.yml @@ -54,7 +54,7 @@ - name: Fail if matrix_playbook_reverse_proxy_type is set incorrectly ansible.builtin.fail: msg: "Detected that variable matrix_playbook_reverse_proxy_type (current value: `{{ matrix_playbook_reverse_proxy_type }}`) appears to be set incorrectly. See roles/custom/matrix-base/defaults/main.yml for valid choices." - when: matrix_playbook_reverse_proxy_type not in ['playbook-installed-traefik', 'playbook-installed-nginx', 'other-traefik-container', 'other-nginx-non-container', 'other-on-same-host', 'other-on-another-host', 'none'] + when: matrix_playbook_reverse_proxy_type not in ['playbook-managed-traefik', 'playbook-managed-nginx', 'other-traefik-container', 'other-nginx-non-container', 'other-on-same-host', 'other-on-another-host', 'none'] - name: Fail if uppercase domain used ansible.builtin.fail: From f37a7a21f17ff8b1b7c7f38eea3f63ecd2dfc04f Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 11 Feb 2023 08:53:32 +0200 Subject: [PATCH 112/382] Delay Postmoogle startup to help Traefik-based setups --- group_vars/matrix_servers | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 5a79ccdc..5b206d91 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -199,6 +199,9 @@ matrix_homeserver_app_service_config_files_auto: | # - so that, when the reverse-proxy is up (Matrix is up), all bots and bridges can be interacted with # - monitoring services (Prometheus, Grafana, ..) get a level of 4000 - they can start later than all-of-Matrix # - services which aren't time-sensitive (various crons and timers) get a level of 5000 - they can start later than all-of-Matrix +# +# `matrix-bot-postmoogle.service` has a higher priority number (4000), unlike other bots' priority (2200), +# because it requires SSL certificates. If Traefik is used, then Postmoogle needs to start at least after Traefik (3500) and Traefik certs dumper (3500). devture_systemd_service_manager_services_list_auto: | {{ ([{'name': 'matrix-backup-borg.timer', 'priority': 5000, 'groups': ['matrix', 'backup', 'borg']}] if matrix_backup_borg_enabled else []) @@ -219,7 +222,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-bot-draupnir.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'draupnir']}] if matrix_bot_draupnir_enabled else []) + - ([{'name': 'matrix-bot-postmoogle.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'postmoogle']}] if matrix_bot_postmoogle_enabled else []) + ([{'name': 'matrix-bot-postmoogle.service', 'priority': 4000, 'groups': ['matrix', 'bots', 'postmoogle']}] if matrix_bot_postmoogle_enabled else []) + ([{'name': 'matrix-bot-chatgpt.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'chatgpt']}] if matrix_bot_chatgpt_enabled else []) + From 94be74e633e8141d3cbe784537d4dcc8f59d9d9e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 11 Feb 2023 08:54:07 +0200 Subject: [PATCH 113/382] Improve traefik-certs-dumper defaults for other-traefik-container setups We'd like to auto-enable traefik-certs-dumper for these setups. `devture_traefik_certs_dumper_ssl_dir_path` will be empty though, so the role's validation will point people in the right direction. --- group_vars/matrix_servers | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 5b206d91..df095b88 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3574,7 +3574,7 @@ devture_traefik_additional_domains_to_obtain_certificates_for: "{{ matrix_ssl_ad # To completely disable the Traefik certs dumper role from running, use `matrix_playbook_traefik_certs_dumper_role_enabled: false`. # See the comment there for more details about why we have both `devture_traefik_certs_dumper_enabled` and `matrix_playbook_traefik_certs_dumper_role_enabled`. -devture_traefik_certs_dumper_enabled: "{{ matrix_playbook_traefik_certs_dumper_role_enabled and devture_traefik_enabled }}" +devture_traefik_certs_dumper_enabled: "{{ matrix_playbook_traefik_certs_dumper_role_enabled }}" devture_traefik_certs_dumper_identifier: matrix-traefik-certs-dumper @@ -3583,7 +3583,7 @@ devture_traefik_certs_dumper_base_path: "{{ matrix_base_data_path }}/traefik-cer devture_traefik_certs_dumper_uid: "{{ matrix_user_uid }}" devture_traefik_certs_dumper_gid: "{{ matrix_user_gid }}" -devture_traefik_certs_dumper_ssl_dir_path: "{{ devture_traefik_ssl_dir_path }}" +devture_traefik_certs_dumper_ssl_dir_path: "{{ devture_traefik_ssl_dir_path if devture_traefik_enabled else '' }}" ######################################################################## # # From 6b0650641ba27b6a13561199c89fcb75b6a2eeb5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 11 Feb 2023 08:58:53 +0200 Subject: [PATCH 114/382] Update matrix_playbook_reverse_proxy_type documentation --- roles/custom/matrix-base/defaults/main.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 8cf63b9b..6a990171 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -275,18 +275,20 @@ matrix_homeserver_app_service_config_files_auto: [] # - `playbook-managed-traefik` # - the playbook will install devture-traefik # - Traefik will do SSL termination, unless you disable it (e.g. `devture_traefik_config_entrypoint_web_secure_enabled: false`) +# - if SSL termination is enabled (as it is by default), you need to populate: `devture_traefik_ssl_email_address` # - it will also install matrix-nginx-proxy in local-only mode, while we migrate the rest of the services to a Traefik-native mode of working # # - `playbook-managed-nginx` # - the playbook will install matrix-nginx-proxy # - matrix-nginx-proxy will do SSL termination with Certbot, unless you change that (see `matrix_ssl_retrieval_method`) +# - if SSL termination is enabled (as it is by default), you need to populate: `matrix_ssl_lets_encrypt_support_email` # # - `other-traefik-container` # - this playbook will not install Traefik # - nevertheless, the playbook expects that you would install Traefik yourself via other means # - you should make sure your Traefik configuration is compatible with what the playbook would have configured (web, web-secure, matrix-federation entrypoints, etc.) # - you need to set `matrix_playbook_reverse_proxyable_services_additional_network` to the name of your Traefik network -# - you may wish to enable `devture_traefik_certs_dumper_enabled` and point it to your Traefik's SSL certificates (`devture_traefik_certs_dumper_ssl_dir_path`) +# - Traefik certs dumper will be enabled by default (`devture_traefik_certs_dumper_enabled`). You need to point it to your Traefik's SSL certificates (`devture_traefik_certs_dumper_ssl_dir_path`) # # - `other-nginx-non-container` # - the playbook will not install matrix-nginx-proxy From be471250dd0c053e161a1ab8449f70eb51c07725 Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Sat, 11 Feb 2023 17:58:19 +0100 Subject: [PATCH 115/382] Move services that crash without hs connectivity to after proxy. --- group_vars/matrix_servers | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index fe83d466..00e63b79 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -187,9 +187,9 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-bot-maubot.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'maubot']}] if matrix_bot_maubot_enabled else []) + - ([{'name': 'matrix-bot-mjolnir.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'mjolnir']}] if matrix_bot_mjolnir_enabled else []) + ([{'name': 'matrix-bot-mjolnir.service', 'priority': 4000, 'groups': ['matrix', 'bots', 'mjolnir']}] if matrix_bot_mjolnir_enabled else []) + - ([{'name': 'matrix-bot-draupnir.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'draupnir']}] if matrix_bot_draupnir_enabled else []) + ([{'name': 'matrix-bot-draupnir.service', 'priority': 4000, 'groups': ['matrix', 'bots', 'draupnir']}] if matrix_bot_draupnir_enabled else []) + ([{'name': 'matrix-bot-postmoogle.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'postmoogle']}] if matrix_bot_postmoogle_enabled else []) + @@ -267,7 +267,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-coturn-reload.timer', 'priority': 5000, 'groups': ['matrix', 'coturn']}] if (matrix_coturn_enabled and matrix_coturn_tls_enabled) else []) + - ([{'name': 'matrix-dimension.service', 'priority': 2500, 'groups': ['matrix', 'integration-managers', 'dimension']}] if matrix_dimension_enabled else []) + ([{'name': 'matrix-dimension.service', 'priority': 4000, 'groups': ['matrix', 'integration-managers', 'dimension']}] if matrix_dimension_enabled else []) + ([{'name': 'matrix-dynamic-dns.service', 'priority': 5000, 'groups': ['matrix', 'dynamic-dns']}] if matrix_dynamic_dns_enabled else []) + From cba63bd4b98b8d030b37787611f8ddf36e96d254 Mon Sep 17 00:00:00 2001 From: Catalan Lover <48515417+FSG-Cat@users.noreply.github.com> Date: Sat, 11 Feb 2023 23:51:13 +0100 Subject: [PATCH 116/382] Upgrade Drapunir from 1.8.0 Beta to 1.8.0 release. --- roles/custom/matrix-bot-draupnir/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-draupnir/defaults/main.yml b/roles/custom/matrix-bot-draupnir/defaults/main.yml index f0aa4da8..5900f21b 100644 --- a/roles/custom/matrix-bot-draupnir/defaults/main.yml +++ b/roles/custom/matrix-bot-draupnir/defaults/main.yml @@ -4,7 +4,7 @@ matrix_bot_draupnir_enabled: true -matrix_bot_draupnir_version: "v1.80.0-beta.0" +matrix_bot_draupnir_version: "v1.80.0" matrix_bot_draupnir_container_image_self_build: false matrix_bot_draupnir_container_image_self_build_repo: "https://github.com/Gnuxie/Draupnir.git" From 17e3c2157e2f74a8f316e6dbb9d278f91ea26a8d Mon Sep 17 00:00:00 2001 From: JokerGermany <30293477+JokerGermany@users.noreply.github.com> Date: Sun, 12 Feb 2023 01:07:51 +0100 Subject: [PATCH 117/382] Add Draupnir to the Readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 94105841..fa906437 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,7 @@ Bots provide various additional functionality to your installation. | [Postmoogle](https://gitlab.com/etke.cc/postmoogle) | x | Email to matrix bot | [Link](docs/configuring-playbook-bot-postmoogle.md) | | [Go-NEB](https://github.com/matrix-org/go-neb) | x | A multi functional bot written in Go | [Link](docs/configuring-playbook-bot-go-neb.md) | | [Mjolnir](https://github.com/matrix-org/mjolnir) | x | A moderation tool for Matrix | [Link](docs/configuring-playbook-bot-mjolnir.md) | +| [Draupnir](https://github.com/Gnuxie/Draupnir) | x | A moderation tool for Matrix (Fork of Mjolnir) | [Link](docs/configuring-playbook-bot-draupnir.md) | | [Buscarron](https://gitlab.com/etke.cc/buscarron) | x | Web forms (HTTP POST) to matrix | [Link](docs/configuring-playbook-bot-buscarron.md) | | [matrix-chatgpt-bot](https://github.com/matrixgpt/matrix-chatgpt-bot) | x | ChatGPT from matrix | [Link](docs/configuring-playbook-bot-chatgpt.md) | From b3f6436a0d182918f24ac74d8525aa88d0c0a2cd Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Feb 2023 15:50:13 +0200 Subject: [PATCH 118/382] Do not enable the Traefik role when reverse-proxy = other-traefik-container --- group_vars/matrix_servers | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index df095b88..72633c6e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -30,7 +30,7 @@ matrix_playbook_docker_installation_enabled: true # In such cases, you'd like to disable the role in this playbook from bothering with Traefik at all (`matrix_playbook_traefik_role_enabled: false`). # If you used `devture_traefik_enabled: false` + `matrix_playbook_traefik_role_enabled: true` instead, you'd see the Treafik role here # try to delete Traefik data (`/devture-traefik`) installed by the other playbook. -matrix_playbook_traefik_role_enabled: true +matrix_playbook_traefik_role_enabled: "{{ matrix_playbook_reverse_proxy_type != 'other-traefik-container' }}" # Controls whether to attach Traefik labels to services. # This is separate from `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`, From 73435dfe100d1765e007bee97ab747c51e5cfcdd Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Feb 2023 15:53:11 +0200 Subject: [PATCH 119/382] Announce Traefik support --- CHANGELOG.md | 108 +++++++++++++++++++++ docs/configuring-playbook-own-webserver.md | 2 + 2 files changed, 110 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 257f57d9..d3f74e70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,111 @@ +# 2023-02-12 + +## Reverse-proxy configuration changes and initial Traefik support + +**TLDR**: + +- there's a new `matrix_playbook_reverse_proxy_type` variable (see [roles/custom/matrix-base/defaults/main.yml](roles/custom/matrix-base/defaults/main.yml)), which lets you tell the playbook what reverse-proxy setup you'd like to have. This makes it easier for people who want to do reverse-proxying in other ways. +- the default reverse-proxy (`matrix_playbook_reverse_proxy_type`) is still `playbook-managed-nginx` (via `matrix-nginx-proxy`), for now. **Existing users should not observe any changes** and can stay on this for now. +- we now have **optional [Traefik](https://traefik.io/) support**, so you could easily host Matrix and other Traefik-native services in containers on the same server. Traefik support is still experimental (albeit, good enough) and will improve over time. It does work, but certain esoteric features may not be there yet. +- **Traefik will become the default reverse-proxy in the near future**. `matrix-nginx-proxy` will either remain as an option, or be completely removed to simplify the playbook + +### Motivation for redoing our reverse-proxy setup + +The playbook has supported various reverse-proxy setups for a long time. +We have various configuration variables (`matrix_nginx_proxy_enabled`, various `_host_bind_port` variables, etc.) which allow the playbook to adapt to these different setups. The whole situation was messy though - hard to figure out and with lots of variables to toggle to make things work as you'd expect - huge **operational complexity**. + +We love containers, proven by the fact that **everything** that this playbook manages runs in a container. Yet, we weren't allowing people to easily host other web-exposed containers alongside Matrix services on the same server. We were using `matrix-nginx-proxy` (our integrated [nginx](https://nginx.org/) server), which was handling web-exposure and SSL termination for our own services, but we **weren't helping you with all your other containers**. + +People who were **using `matrix-nginx-proxy`** were on the happy path on which everything worked well by default (Matrix-wise), **but** could not easily run other web-exposed services on their Matrix server because `matrix-nginx-proxy` was occupying ports `80` and `443`. Other services which wanted to get web exposure either had to be plugged into `matrix-nginx-proxy` (somewhat difficult) or people had to forgo using `matrix-nginx-proxy` in favor of something else. + +Of those that decided to forgo `matrix-nginx-proxy`, many were **using nginx** on the same server without a container. This was likely some ancient nginx version, depending on your choice of distro. The Matrix playbook was trying to be helpful and even with `matrix_nginx_proxy_enabled: false` was still generating nginx configuration in `/matrix/nginx-proxy/conf.d`. Those configuration files were adapted for inclusion into an nginx server running locally. Disabling the `matrix-nginx-proxy` role like this, yet still having it produce files is a bit disgusting, but it's what we've had since the early beginnings of this playbook. + +Others still, wanted to run Matrix locally (no SSL certificates), regardless of which web server technology this relied on, and then **reverse-proxy from another machine on the network** which was doing SSL termination. These people were: + +- *either* relying on `matrix_nginx_proxy_enabled: false` as well, combined with exposing services manually (setting `_bind_port` variables) +- *or* better yet, they were keeping `matrix-nginx-proxy` enabled, but in `http`-only mode (no SSL certificate retrieval). + +Despite this operational complexity, things worked and were reasonably flexible to adapt to all these situations. + +When using `matrix-nginx-proxy` as is, we still had another problem - one of **internal playbook complexity**. Too many services need to be web-exposed (port 80/443, SSL certificates). Because of this, they all had to integrate with the `matrix-nginx-proxy` role. Tens of different roles explicitly integrating with `matrix-nginx-proxy` is not what we call clean. The `matrix-nginx-proxy` role contains variables for many of these roles (yikes). Other roles were more decoupled from it and were injecting configuration into `matrix-nginx-proxy` at runtime - see all the `inject_into_nginx_proxy.yml` task files in this playbook (more decoupled, but still.. yikes). + +The next problem is one of **efficiency, interoperability and cost-saving**. We're working on other playbooks: + +- [vaultwarden-docker-ansible-deploy](https://github.com/spantaleev/vaultwarden-docker-ansible-deploy) for hosting the [Vaultwarden](https://github.com/dani-garcia/vaultwarden) server - an alternative implementation of the [Bitwarden](https://bitwarden.com/) password manager +- [gitea-docker-ansible-deploy](https://github.com/spantaleev/gitea-docker-ansible-deploy) - for hosting the [Gitea](https://gitea.io/) git source code hosting service +- [nextcloud-docker-ansible-deploy](https://github.com/spantaleev/nextcloud-docker-ansible-deploy) - for hosting the [Nextcloud](https://nextcloud.com/) groupware platform + +We'd love for users to be able to **seamlessly use all these playbooks (and others, even) against a single server**. We don't want `matrix-nginx-proxy` to have a monopoly on port `80`/`443` and make it hard for other services to join in on the party. Such a thing forces people into running multiple servers (one for each service), which does provide nice security benefits, but is costly and ineffiecient. We'd like to make self-hosting these services cheap and easy. + +These other playbooks have been using [Traefik](https://traefik.io/) as their default reverse-proxy for a long time. They can all coexist nicely together (as an example, see the [Interoperability](https://github.com/spantaleev/nextcloud-docker-ansible-deploy/blob/master/docs/configuring-playbook-interoperability.md) documentation for the [Nextcloud playbook](https://github.com/spantaleev/nextcloud-docker-ansible-deploy)). Now that this playbook is gaining Traefik support, it will be able to interoperate with them. If you're going this way, make sure to have the Matrix playbook install Traefik and have the others use `*_reverse_proxy_type: other-traefik-container`. + +Finally, at [etke.cc - a managed Matrix server hosting service](https://etke.cc) (built on top of this playbook, and coincidentally [turning 2 years old today](https://etke.cc/news/upsyw4ykbtgmwhz8k7ukldx0zbbfq-fh0iqi3llixi0/) 🎉), we're allowing people to host some additional services besides Matrix components. Exposing these services to the web requires ugly hacks and configuration files being dropped into `/matrix/nginx-proxy/conf.d`. We believe that everything should run in independent containers and be exposed to the web via a Traefik server, without a huge Ansible role like `matrix-nginx-proxy` that everything else needs to integrate with. + +### How do these changes fix all these problems? + +The new `matrix_playbook_reverse_proxy_type` lets you easily specify your preferred reverse-proxy type, including `other-on-same-host`, `other-on-another-host` and `none`, so people who'd like to reverse-proxy with their own web server have more options now. + +Using Traefik greatly simplifies things, so going forward we'll have a simpler and easier to maintain playbook, which is also interoperable with other services. + +Traefik is a web server, which has been specifically **designed for reverse-proxying to services running in containers**. It's ideal for usage in an Ansible playbook which runs everything in containers. + +**Traefik obtains SSL certificates automatically**, so there's no need for plugging additional tools like [Certbot](https://certbot.eff.org/) into your web server (like we were doing in the `matrix-nginx-proxy` role). No more certificate renewal timers, web server reloading timers, etc. It's just simpler. + +Traefik is a **modern web server**. [HTTP/3](https://doc.traefik.io/traefik/routing/entrypoints/#http3) is supported already (experimentally) and will move to stable soon, in the upcoming Traefik v3 release. + +Traefik does not lock important functionality we'd like to use into [plus packages like nginx does](https://www.nginx.com/products/nginx/), leading us to resolve to configuration workarounds. The default Traefik package is good enough as it is. + +### Where we're at right now? + +`matrix_playbook_reverse_proxy_type` still defaults to a value of `playbook-managed-nginx`. + +Unless we have some regression, **existing users should be able to update their Matrix server and not observe any changes**. Their setup should still remain on nginx and everything should still work as expected. + +#### How do I explicitly switch to Traefik right now? + +**Users who want to migrate to Traefik** today, can do so by using configuration like this: + +```yaml +matrix_playbook_reverse_proxy_type: playbook-managed-traefik + +devture_traefik_ssl_email_address: YOUR_EMAIL_ADDRESS +``` + +Switching to Traefik will obtain new SSL certificates from Let's Encrypt (stored in `/devture-traefik/ssl/acme.json`). **The switch is reversible**. You can always go back to `playbook-managed-nginx` if Traefik is causing you trouble. + +**Note**: toggling `matrix_playbook_reverse_proxy_type` between Traefik and nginx will uninstall the Traefik role and all of its data (under `/devture-traefik`), so you may run into a Let's Encrypt rate limit if you do it often. + +Treafik directly reverse-proxies to **some** services right now, but for most other services it goes through `matrix-nginx-proxy` (e.g. Traefik -> `matrix-nginx-proxy` -> [Ntfy](docs/configuring-playbook-ntfy.md)). So, even if you opt into Traefik, you'll still see `matrix-nginx-proxy` being installed in local-only mode. This will improve with time. + +Some services (like [Coturn](docs/configuring-playbook-turn.md) and [Postmoogle](docs/configuring-playbook-bot-postmoogle.md)) cannot be reverse-proxied to directly from Traefik, so they require direct access to SSL certificate files extracted out of Traefik. The playbook does this automatically thanks to a new [com.devture.ansible.role.traefik_certs_dumper](https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper) role utilizing the [traefik-certs-dumper](https://github.com/ldez/traefik-certs-dumper) tool. + +Our Traefik setup mostly works, but certain esoteric features may not work. If you have a default setup, we expect you to have a good experience. + + +### Where we're going in the near future? + +The `matrix-nginx-proxy` role is quite messy. It manages both nginx and Certbot and its certificate renewal scripts and timers. It generates configuration even when the role is disabled (weird). Although it doesn't directly reach into variables from other roles, it has explicit awareness of various other services that it reverse-proxies to (`roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-ntfy.conf.j2`, etc.). We'd like to clean this up. The only way is probably to just get rid of the whole thing at some point. + +For now, `matrix-nginx-proxy` will stay around. + +As mentioned above, Traefik still reverse-proxies to some (most) services by going through a local-only `matrix-nginx-proxy` server. This has allowed us to add Traefik support to the playbook early on (without having to rework all services), but is not the final goal. We'll **work on making each service support Traefik natively**, so that traffic will not need to go through `matrix-nginx-proxy` anymore. In the end, choosing Traefik should only give you a pure Traefik installation with no `matrix-nginx-proxy` in sight. + +As Traefik support becomes complete and proves to be stable for a while, especially as a playbook default, we will **most likely remove `matrix-nginx-proxy` completely**. It will likely be some months before this happens though. Keeping support for both Traefik and nginx in the playbook will be a burden, especially with most of us running Traefik in the future. The Traefik role should do everything nginx does in a better and cleaner way. Users who use their own `nginx` server on the Matrix server will be inconvenienced, as nothing will generate ready-to-include nginx configuration for them. Still, we hope it won't be too hard to migrate their setup to another way of doing things, like: + +- not using nginx anymore. A common reason for using nginx until now was that you were running other containers and you need your own nginx to reverse-proxy to all of them. Just switch them to Traefik as well. +- running Traefik in local-only mode (`devture_traefik_config_entrypoint_web_secure_enabled: false`) and using some nginx configuration which reverse-proxies to Traefik (we should introduce examples for this in `examples/nginx`). + +### How do I help? + +You can help by: + +- **explicitly switching your server to Traefik** right now (see example configuration in [How do I explicitly switch to Traefik right now?](#how-do-i-explicitly-switch-to-traefik-right-now) above), testing, reporting troubles + +- **adding native Traefik support to a role** (requires adding Traefik labels, etc.) - for inspiration, see these roles ([prometheus_node_exporter](https://gitlab.com/etke.cc/roles/prometheus_node_exporter), [prometheus_postgres_exporter](https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter)) and how they're hooked into the playbook via [group_vars/matrix_servers](group_vars/matrix_servers). + +- **adding reverse-proxying examples for nginx users** in `examples/nginx`. People who insist on using their own `nginx` server on the same Matrix host, can run Traefik in local-only mode (`devture_traefik_config_entrypoint_web_secure_enabled: false`) and reverse-proxy to the Traefik server + + # 2023-02-10 ## Matrix Authentication Support for Jitsi diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 06f21b32..e978a1e1 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -1,5 +1,7 @@ # Using your own webserver, instead of this playbook's nginx proxy (optional, advanced) +**Note**: the playbook is [in the process of moving to Traefik](../CHANGELOG.md#reverse-proxy-configuration-changes-and-initial-traefik-support). The documentation below should be correct, but things will change soon. + By default, this playbook installs its own nginx webserver (called `matrix-nginx-proxy`, in a Docker container) which listens on ports 80 and 443. If that's alright, you can skip this. From bc4cb7bb6a52185b43fe710bfa7ec2e956981c55 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Feb 2023 16:03:44 +0200 Subject: [PATCH 120/382] Update configuring-playbook-own-webserver.md with matrix_playbook_reverse_proxy_type awareness --- docs/configuring-playbook-own-webserver.md | 89 +++++++--------------- 1 file changed, 26 insertions(+), 63 deletions(-) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index e978a1e1..77fa018d 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -102,13 +102,7 @@ You can disable such behavior and make the integrated nginx reverse-proxy webser You would need some configuration like this: ```yaml -# Do not retrieve SSL certificates. This shall be managed by another webserver or other means. -matrix_ssl_retrieval_method: none - -# Do not try to serve HTTPS, since we have no SSL certificates. -# Disabling this also means services will be served on the HTTP port -# (`matrix_nginx_proxy_container_http_host_bind_port`). -matrix_nginx_proxy_https_enabled: false +matrix_playbook_reverse_proxy_type: other-on-same-host # Do not listen for HTTP on port 80 globally (default), listen on the loopback interface. # If you'd like, you can make it use the local network as well and reverse-proxy from another local machine. @@ -122,15 +116,9 @@ matrix_nginx_proxy_container_http_host_bind_port: '127.0.0.1:81' matrix_nginx_proxy_container_federation_host_bind_port: '127.0.0.1:8449' # Coturn relies on SSL certificates that have already been obtained. -# Since we don't obtain any certificates (`matrix_ssl_retrieval_method: none` above), it won't work by default. +# Since we don't obtain any certificates in this `other-on-same-host` mode, it won't work by default. # An alternative is to tweak some of: `matrix_coturn_tls_enabled`, `matrix_coturn_tls_cert_path` and `matrix_coturn_tls_key_path`. matrix_coturn_enabled: false - -# Trust the reverse proxy to send the correct `X-Forwarded-Proto` header as it is handling the SSL connection. -matrix_nginx_proxy_trust_forwarded_proto: true - -# Trust and use the other reverse proxy's `X-Forwarded-For` header. -matrix_nginx_proxy_x_forwarded_for: '$proxy_add_x_forwarded_for' ``` With this, nginx would still be in use, but it would not bother with anything SSL related or with taking up public ports. @@ -142,63 +130,38 @@ The expected domains vary depending on the services you have enabled (`matrix.DO ### Sample configuration for running behind Traefik 2.0 -Below is a sample configuration for using this playbook with a [Traefik](https://traefik.io/) 2.0 reverse proxy. +To run behind Traefik, you can: -```yaml -# Disable generation and retrieval of SSL certs -matrix_ssl_retrieval_method: none +- use a [Playbook-managed Traefik installation](#playbook-managed-traefik-installation) +- or, use [Your own Traefik server (not managed by the playbook)](#your-own-traefik-server-not-managed-by-the-playbook) -# Configure Nginx to only use plain HTTP -matrix_nginx_proxy_https_enabled: false +#### Playbook-managed Traefik installation -# Don't bind any HTTP or federation port to the host -# (Traefik will proxy directly into the containers) -matrix_nginx_proxy_container_http_host_bind_port: '' -matrix_nginx_proxy_container_federation_host_bind_port: '' +The playbook can install and manage Traefik for you using the [com.devture.ansible.role.traefik](https://github.com/devture/com.devture.ansible.role.traefik) role. -# Trust the reverse proxy to send the correct `X-Forwarded-Proto` header as it is handling the SSL connection. -matrix_nginx_proxy_trust_forwarded_proto: true +It's simplest if you go with this method. You will need the following configuration: -# Trust and use the other reverse proxy's `X-Forwarded-For` header. -matrix_nginx_proxy_x_forwarded_for: '$proxy_add_x_forwarded_for' +```yaml +matrix_playbook_reverse_proxy_type: playbook-managed-traefik -# Disable Coturn because it needs SSL certs -# (Clients can, though exposing IP address, use Matrix.org TURN) -matrix_coturn_enabled: false +devture_traefik_ssl_email_address: YOUR_EMAIL_ADDRESS +``` + +#### Your own Traefik server (not managed by the playbook) -# All containers need to be on the same Docker network as Traefik -# (This network should already exist and Traefik should be using this network) -matrix_docker_network: 'traefik' - -matrix_nginx_proxy_container_extra_arguments: - # May be unnecessary depending on Traefik config, but can't hurt - - '--label "traefik.enable=true"' - - # The Nginx proxy container will receive traffic from these subdomains - - '--label "traefik.http.routers.matrix-nginx-proxy.rule=Host(`{{ matrix_server_fqn_matrix }}`,`{{ matrix_server_fqn_element }}`,`{{ matrix_server_fqn_dimension }}`,`{{ matrix_server_fqn_jitsi }}`)"' - # (The 'web-secure' entrypoint must bind to port 443 in Traefik config) - - '--label "traefik.http.routers.matrix-nginx-proxy.entrypoints=web-secure"' - # (The 'default' certificate resolver must be defined in Traefik config) - - '--label "traefik.http.routers.matrix-nginx-proxy.tls.certResolver=default"' - # Traefik requires that we declare which service this router is using - - '--label "traefik.http.routers.matrix-nginx-proxy.service=matrix-nginx-proxy"' - # The Nginx proxy container uses port 8080 internally - - '--label "traefik.http.services.matrix-nginx-proxy.loadbalancer.server.port=8080"' - - # Federation - - '--label "traefik.http.routers.matrix-nginx-proxy-federation.rule=Host(`{{ matrix_server_fqn_matrix }}`)"' - # (The 'federation' entrypoint must bind to port 8448 in Traefik config) - - '--label "traefik.http.routers.matrix-nginx-proxy-federation.entrypoints=federation"' - # (The 'default' certificate resolver must be defined in Traefik config) - - '--label "traefik.http.routers.matrix-nginx-proxy-federation.tls.certResolver=default"' - # Traefik requires that we declare which service this router is using - - '--label "traefik.http.routers.matrix-nginx-proxy-federation.service=matrix-nginx-proxy-federation"' - # The Nginx proxy container uses port `matrix_nginx_proxy_proxy_matrix_federation_port (8448) internally - - '--label "traefik.http.services.matrix-nginx-proxy-federation.loadbalancer.server.port={{ matrix_nginx_proxy_proxy_matrix_federation_port }}"' - - '--label "traefik.http.services.matrix-nginx-proxy-federation.loadbalancer.server.scheme={{ "https" if matrix_nginx_proxy_https_enabled else "http" }}"' +If you'd like to run Traefik yourself, you can use configuration like this: + +```yaml +matrix_playbook_reverse_proxy_type: other-traefik-container + +matrix_playbook_reverse_proxyable_services_additional_network: your-traefik-network ``` -This method uses labels attached to the Nginx and Synapse containers to provide the Traefik Docker provider with the information it needs to proxy `matrix.DOMAIN`, `element.DOMAIN`, `dimension.DOMAIN` and `jitsi.DOMAIN`. Some [static configuration](https://docs.traefik.io/v2.0/reference/static-configuration/file/) is required in Traefik; namely, having endpoints on ports 443 and 8448 and having a certificate resolver. +In this mode all roles will still have Traefik labels attached. You will, however, need to configure your Traefik instance and its entrypoints. + +By default, the playbook congiures services use a `web-secure` (443) and `matrix-federation` (8448) entrypoints, as well as a `default` certificate resolver. + +Below is some configuration for running Traefik yourself (although we recommend using a [playbook-managed Traefik installation](#playbook-managed-traefik-installation)). Note that this configuration on its own does **not** redirect traffic on port 80 (plain HTTP) to port 443 for HTTPS, which may cause some issues, since the built-in Nginx proxy usually does this. If you are not already doing this in Traefik, it can be added to Traefik in a [file provider](https://docs.traefik.io/v2.0/providers/file/) as follows: @@ -228,7 +191,7 @@ version: "3.3" services: traefik: - image: "traefik:v2.3" + image: "docker.io/traefik:v2.9.6" restart: always container_name: "traefik" networks: From 972043cfaf5795fa8c3a82ab974bd817c87977c1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Feb 2023 17:14:25 +0200 Subject: [PATCH 121/382] Fix trying to start devture-traefik when not necessarily enabled Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2465 --- group_vars/matrix_servers | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 46c80874..1b31bf00 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -356,7 +356,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-user-verification-service.service', 'priority': 800, 'groups': ['matrix', 'matrix-user-verification-service']}] if matrix_user_verification_service_enabled else []) + - ([{'name': 'devture-traefik.service', 'priority': 3000, 'groups': ['matrix', 'traefik', 'reverse-proxies']}] if matrix_playbook_traefik_role_enabled else []) + ([{'name': 'devture-traefik.service', 'priority': 3000, 'groups': ['matrix', 'traefik', 'reverse-proxies']}] if devture_traefik_enabled else []) + ([{'name': (devture_traefik_certs_dumper_identifier + '.service'), 'priority': 3500, 'groups': ['matrix', 'traefik-certs-dumper']}] if matrix_playbook_traefik_certs_dumper_role_enabled else []) }} From 79413e7717b669f07cdc32909aecbcb6114c1e6e Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Sun, 12 Feb 2023 13:09:53 -0500 Subject: [PATCH 122/382] updated dendrite --- roles/custom/matrix-dendrite/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 5e6d7d3b..525f4854 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -6,7 +6,7 @@ matrix_dendrite_enabled: true matrix_dendrite_docker_image: "{{ matrix_dendrite_docker_image_name_prefix }}matrixdotorg/dendrite-monolith:{{ matrix_dendrite_docker_image_tag }}" matrix_dendrite_docker_image_name_prefix: "docker.io/" -matrix_dendrite_docker_image_tag: "v0.11.0" +matrix_dendrite_docker_image_tag: "v0.11.1" matrix_dendrite_docker_image_force_pull: "{{ matrix_dendrite_docker_image.endswith(':latest') }}" matrix_dendrite_base_path: "{{ matrix_base_data_path }}/dendrite" From 6939a3d6d319c896b525a70c1b4ab697968e7458 Mon Sep 17 00:00:00 2001 From: Jayesh Nirve Date: Mon, 13 Feb 2023 12:06:20 +0530 Subject: [PATCH 123/382] fix: only add element related entries to client well-known if element is enabled (#2453) * fix: only add element related entries to client well-known if element is enabled * Fix matrix-base/defaults/main.yml syntax --------- Co-authored-by: Slavi Pantaleev --- roles/custom/matrix-base/defaults/main.yml | 14 +++++++++++--- roles/custom/matrix-base/tasks/validate_config.yml | 3 +++ .../static-files/well-known/matrix-client.j2 | 11 ++++++----- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 6a990171..e72142c3 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -154,17 +154,25 @@ matrix_client_element_jitsi_preferredDomain: '' # noqa var-naming # Controls whether Element should use End-to-End Encryption by default. # Setting this to false will update `/.well-known/matrix/client` and tell Element clients to avoid E2EE. # See: https://github.com/vector-im/element-web/blob/develop/docs/e2ee.md -matrix_client_element_e2ee_default: true +matrix_well_known_matrix_client_io_element_e2ee_default: true # Controls whether Element should require a secure backup set up before Element can be used. # Setting this to true will update `/.well-known/matrix/client` and tell Element require a secure backup. # See: https://github.com/vector-im/element-web/blob/develop/docs/e2ee.md -matrix_client_element_e2ee_secure_backup_required: false +matrix_well_known_matrix_client_io_element_e2ee_secure_backup_required: false # Controls which backup methods from ["key", "passphrase"] should be used, both is the default. # Setting this to other then empty will update `/.well-known/matrix/client` and tell Element which method to use # See: https://github.com/vector-im/element-web/blob/develop/docs/e2ee.md -matrix_client_element_e2ee_secure_backup_setup_methods: [] +matrix_well_known_matrix_client_io_element_e2ee_secure_backup_setup_methods: [] + +# Controls whether element related entries should be added to the client well-known. Override this to false to hide +# element related well-known entries. +# By default if any of the following change from their default this is set to true: +# `matrix_well_known_matrix_client_io_element_e2ee_default` +# `matrix_well_known_matrix_client_io_element_e2ee_secure_backup_required` +# `matrix_well_known_matrix_client_io_element_e2ee_secure_backup_setup_methods` +matrix_well_known_matrix_client_io_element_e2ee_entries_enabled: "{{ not matrix_well_known_matrix_client_io_element_e2ee_default or matrix_well_known_matrix_client_io_element_e2ee_secure_backup_required or matrix_well_known_matrix_client_io_element_e2ee_secure_backup_setup_methods | length > 0 }}" # Default `/.well-known/matrix/client` configuration - it covers the generic use case. # You can customize it by controlling the various variables inside the template file that it references. diff --git a/roles/custom/matrix-base/tasks/validate_config.yml b/roles/custom/matrix-base/tasks/validate_config.yml index cd6d20cd..ab06ffcd 100644 --- a/roles/custom/matrix-base/tasks/validate_config.yml +++ b/roles/custom/matrix-base/tasks/validate_config.yml @@ -18,6 +18,9 @@ - {'old': 'hostname_riot', 'new': 'matrix_server_fqn_element'} - {'old': 'matrix_server_fqn_riot', 'new': 'matrix_server_fqn_element'} - {'old': 'matrix_local_bin_path', 'new': ''} + - {'old': 'matrix_client_element_e2ee_default', 'new': 'matrix_well_known_matrix_client_io_element_e2ee_default'} + - {'old': 'matrix_client_element_e2ee_secure_backup_required', 'new': 'matrix_well_known_matrix_client_io_element_e2ee_secure_backup_required'} + - {'old': 'matrix_client_element_e2ee_secure_backup_setup_methods', 'new': 'matrix_well_known_matrix_client_io_element_e2ee_secure_backup_setup_methods'} # We have a dedicated check for this variable, because we'd like to have a custom (friendlier) message. - name: Fail if matrix_homeserver_generic_secret_key is undefined diff --git a/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 b/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 index 4595bed1..63b2c812 100644 --- a/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 +++ b/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 @@ -31,13 +31,14 @@ "map_style_url": "https://{{ matrix_server_fqn_element }}/map_style.json" } {% endif %} - , + {% if matrix_well_known_matrix_client_io_element_e2ee_entries_enabled %}, "io.element.e2ee": { - "default": {{ matrix_client_element_e2ee_default|to_json }}, - "secure_backup_required": {{ matrix_client_element_e2ee_secure_backup_required|to_json }}, - "secure_backup_setup_methods": {{ matrix_client_element_e2ee_secure_backup_setup_methods|to_json }} + "default": {{ matrix_well_known_matrix_client_io_element_e2ee_default|to_json }}, + "secure_backup_required": {{ matrix_well_known_matrix_client_io_element_e2ee_secure_backup_required|to_json }}, + "secure_backup_setup_methods": {{ matrix_well_known_matrix_client_io_element_e2ee_secure_backup_setup_methods|to_json }} }, + {% if matrix_well_known_matrix_client_io_element_e2ee_entries_enabled %}, "im.vector.riot.e2ee": { - "default": {{ matrix_client_element_e2ee_default|to_json }} + "default": {{ matrix_well_known_matrix_client_io_element_e2ee_default|to_json }} } } From af10d350bc3bc16c3cd59d60f65c50a2f2b9d887 Mon Sep 17 00:00:00 2001 From: td Date: Mon, 13 Feb 2023 12:32:43 +0530 Subject: [PATCH 124/382] fix: missing endif in client well-known --- .../templates/static-files/well-known/matrix-client.j2 | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 b/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 index 63b2c812..45fc9b66 100644 --- a/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 +++ b/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 @@ -36,9 +36,11 @@ "default": {{ matrix_well_known_matrix_client_io_element_e2ee_default|to_json }}, "secure_backup_required": {{ matrix_well_known_matrix_client_io_element_e2ee_secure_backup_required|to_json }}, "secure_backup_setup_methods": {{ matrix_well_known_matrix_client_io_element_e2ee_secure_backup_setup_methods|to_json }} - }, + } + {% endif %} {% if matrix_well_known_matrix_client_io_element_e2ee_entries_enabled %}, "im.vector.riot.e2ee": { "default": {{ matrix_well_known_matrix_client_io_element_e2ee_default|to_json }} } + {% endif %} } From 78c35136b2cff4a6f24d01077c2fc19c36f1a878 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 10:53:11 +0200 Subject: [PATCH 125/382] Replace matrix-backup-borg with an external role --- docs/configuring-playbook-backup-borg.md | 18 +-- group_vars/matrix_servers | 38 ++++--- playbooks/matrix.yml | 2 +- requirements.yml | 3 + .../matrix-backup-borg/defaults/main.yml | 107 ------------------ .../custom/matrix-backup-borg/tasks/main.yml | 20 ---- .../tasks/setup_install.yml | 107 ------------------ .../tasks/setup_uninstall.yml | 25 ---- .../tasks/validate_config.yml | 16 --- .../templates/config.yaml.j2 | 43 ------- .../matrix-backup-borg/templates/passwd.j2 | 29 ----- .../matrix-backup-borg/templates/sshkey.j2 | 1 - .../systemd/matrix-backup-borg.service.j2 | 65 ----------- .../systemd/matrix-backup-borg.timer.j2 | 10 -- .../defaults/main.yml | 3 + .../tasks/validate_config.yml | 17 +++ 16 files changed, 57 insertions(+), 447 deletions(-) delete mode 100644 roles/custom/matrix-backup-borg/defaults/main.yml delete mode 100644 roles/custom/matrix-backup-borg/tasks/main.yml delete mode 100644 roles/custom/matrix-backup-borg/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-backup-borg/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-backup-borg/tasks/validate_config.yml delete mode 100644 roles/custom/matrix-backup-borg/templates/config.yaml.j2 delete mode 100644 roles/custom/matrix-backup-borg/templates/passwd.j2 delete mode 100644 roles/custom/matrix-backup-borg/templates/sshkey.j2 delete mode 100644 roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.service.j2 delete mode 100644 roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.timer.j2 diff --git a/docs/configuring-playbook-backup-borg.md b/docs/configuring-playbook-backup-borg.md index b6b5cb09..2f6ab5ad 100644 --- a/docs/configuring-playbook-backup-borg.md +++ b/docs/configuring-playbook-backup-borg.md @@ -6,9 +6,9 @@ That means your daily incremental backups can be stored in a fraction of the spa You will need a remote server where borg will store the backups. There are hosted, borg compatible solutions available, such as [BorgBase](https://www.borgbase.com). -The backup will run based on `matrix_backup_borg_schedule` var (systemd timer calendar), default: 4am every day. +The backup will run based on `backup_borg_schedule` var (systemd timer calendar), default: 4am every day. -By default, if you're using the integrated Postgres database server (as opposed to [an external Postgres server](configuring-playbook-external-postgres.md)), Borg backups will also include dumps of your Postgres database. An alternative solution for backing up the Postgres database is [postgres backup](configuring-playbook-postgres-backup.md). If you decide to go with another solution, you can disable Postgres-backup support for Borg using the `matrix_backup_borg_postgresql_enabled` variable. +By default, if you're using the integrated Postgres database server (as opposed to [an external Postgres server](configuring-playbook-external-postgres.md)), Borg backups will also include dumps of your Postgres database. An alternative solution for backing up the Postgres database is [postgres backup](configuring-playbook-postgres-backup.md). If you decide to go with another solution, you can disable Postgres-backup support for Borg using the `backup_borg_postgresql_enabled` variable. ## Prerequisites @@ -38,11 +38,11 @@ cat PUBKEY | ssh USER@HOST 'dd of=.ssh/authorized_keys oflag=append conv=notrunc Minimal working configuration (`inventory/host_vars/matrix.DOMAIN/vars.yml`) to enable borg backup: ```yaml -matrix_backup_borg_enabled: true -matrix_backup_borg_location_repositories: +backup_borg_enabled: true +backup_borg_location_repositories: - ssh://USER@HOST/./REPO -matrix_backup_borg_storage_encryption_passphrase: "PASSPHRASE" -matrix_backup_borg_ssh_key_private: | +backup_borg_storage_encryption_passphrase: "PASSPHRASE" +backup_borg_ssh_key_private: | -----BEGIN OPENSSH PRIVATE KEY----- TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZW xpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRv @@ -58,11 +58,11 @@ where: * HOST - SSH host of a provider/server * REPO - borg repository name, it will be initialized on backup start, eg: `matrix`, regarding Syntax see [Remote repositories](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls) * PASSPHRASE - passphrase used for encrypting backups, you may generate it with `pwgen -s 64 1` or use any password manager -* PRIVATE KEY - the content of the **private** part of the SSH key you created before. The whole key (all of its belonging lines) under `matrix_backup_borg_ssh_key_private` needs to be indented with 2 spaces +* PRIVATE KEY - the content of the **private** part of the SSH key you created before. The whole key (all of its belonging lines) under `backup_borg_ssh_key_private` needs to be indented with 2 spaces -To backup without encryption, add `matrix_backup_borg_encryption: 'none'` to your vars. This will also enable the `matrix_backup_borg_unknown_unencrypted_repo_access_is_ok` variable. +To backup without encryption, add `backup_borg_encryption: 'none'` to your vars. This will also enable the `backup_borg_unknown_unencrypted_repo_access_is_ok` variable. -`matrix_backup_borg_location_source_directories` defines the list of directories to back up: it's set to `{{ matrix_base_data_path }}` by default, which is the base directory for every service's data, such as Synapse, Postgres and the bridges. You might want to exclude certain directories or file patterns from the backup using the `matrix_backup_borg_location_exclude_patterns` variable. +`backup_borg_location_source_directories` defines the list of directories to back up: it's set to `{{ matrix_base_data_path }}` by default, which is the base directory for every service's data, such as Synapse, Postgres and the bridges. You might want to exclude certain directories or file patterns from the backup using the `backup_borg_location_exclude_patterns` variable. Check the `roles/custom/matrix-backup-borg/defaults/main.yml` file for the full list of available options. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 1b31bf00..fbae0304 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -204,7 +204,7 @@ matrix_homeserver_app_service_config_files_auto: | # because it requires SSL certificates. If Traefik is used, then Postmoogle needs to start at least after Traefik (3500) and Traefik certs dumper (3500). devture_systemd_service_manager_services_list_auto: | {{ - ([{'name': 'matrix-backup-borg.timer', 'priority': 5000, 'groups': ['matrix', 'backup', 'borg']}] if matrix_backup_borg_enabled else []) + ([{'name': (backup_borg_identifier + '.timer'), 'priority': 5000, 'groups': ['matrix', 'backup', 'borg']}] if backup_borg_enabled else []) + ([{'name': 'matrix-bot-buscarron.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'buscarron']}] if matrix_bot_buscarron_enabled else []) + @@ -1871,34 +1871,42 @@ matrix_bot_draupnir_systemd_required_services_list: | # ###################################################################### + ###################################################################### # -# matrix-backup-borg +# etke/backup_borg # ###################################################################### -matrix_backup_borg_enabled: false +backup_borg_enabled: false + +backup_borg_identifier: matrix-backup-borg + +backup_borg_uid: "{{ matrix_user_uid }}" +backup_borg_gid: "{{ matrix_user_gid }}" -matrix_backup_borg_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm32', 'arm64'] }}" +backup_borg_container_network: "{{ devture_postgres_container_network if devture_postgres_enabled else backup_borg_identifier }}" -matrix_backup_borg_postgresql_enabled: "{{ devture_postgres_enabled }}" -matrix_backup_borg_postgresql_databases_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" -matrix_backup_borg_postgresql_databases_username: "{{ devture_postgres_connection_username if devture_postgres_enabled else '' }}" -matrix_backup_borg_postgresql_databases_password: "{{ devture_postgres_connection_password if devture_postgres_enabled else '' }}" -matrix_backup_borg_postgresql_databases_port: "{{ devture_postgres_connection_port if devture_postgres_enabled else 5432 }}" -matrix_backup_borg_postgresql_databases: "{{ devture_postgres_managed_databases | map(attribute='name') if devture_postgres_enabled else [] }}" +backup_borg_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm32', 'arm64'] }}" -matrix_backup_borg_location_source_directories: +backup_borg_postgresql_enabled: "{{ devture_postgres_enabled }}" +backup_borg_postgresql_databases_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" +backup_borg_postgresql_databases_username: "{{ devture_postgres_connection_username if devture_postgres_enabled else '' }}" +backup_borg_postgresql_databases_password: "{{ devture_postgres_connection_password if devture_postgres_enabled else '' }}" +backup_borg_postgresql_databases_port: "{{ devture_postgres_connection_port if devture_postgres_enabled else 5432 }}" +backup_borg_postgresql_databases: "{{ devture_postgres_managed_databases | map(attribute='name') if devture_postgres_enabled else [] }}" + +backup_borg_location_source_directories: - "{{ matrix_base_data_path }}" -matrix_backup_borg_location_exclude_patterns: | +backup_borg_location_exclude_patterns: | {{ ([matrix_synapse_media_store_path + '/local_thumbnails', matrix_synapse_media_store_path + '/remote_thumbnail', matrix_synapse_media_store_path + '/url_cache', matrix_synapse_media_store_path + '/url_cache_thumbnails'] if matrix_homeserver_implementation == 'synapse' else []) + ([devture_postgres_data_path] if devture_postgres_enabled else []) }} -matrix_backup_borg_systemd_required_services_list: | +backup_borg_systemd_required_services_list: | {{ ['docker.service'] + @@ -1907,9 +1915,11 @@ matrix_backup_borg_systemd_required_services_list: | ###################################################################### # -# /matrix-backup-borg +# /etke/backup_borg # ###################################################################### + + ###################################################################### # # matrix-cactus-comments diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 508adb4d..d976a012 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -112,7 +112,7 @@ - role: galaxy/com.devture.ansible.role.postgres_backup - - custom/matrix-backup-borg + - role: galaxy/backup_borg - custom/matrix-user-creator - custom/matrix-common-after diff --git a/requirements.yml b/requirements.yml index 1341bbbe..75449124 100644 --- a/requirements.yml +++ b/requirements.yml @@ -36,6 +36,9 @@ - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.11.1-2 +- src: git+https://gitlab.com/etke.cc/roles/backup_borg.git + version: v1.2.3-1.7.5-0 + - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: fb09fd26f877372417d5586f1e79e83f983f0bd6 diff --git a/roles/custom/matrix-backup-borg/defaults/main.yml b/roles/custom/matrix-backup-borg/defaults/main.yml deleted file mode 100644 index ad5a0281..00000000 --- a/roles/custom/matrix-backup-borg/defaults/main.yml +++ /dev/null @@ -1,107 +0,0 @@ ---- -# Project source code URL: https://gitlab.com/etke.cc/borgmatic - -matrix_backup_borg_enabled: true - -matrix_backup_borg_base_path: "{{ matrix_base_data_path }}/backup-borg" -matrix_backup_borg_config_path: "{{ matrix_backup_borg_base_path }}/config" - -matrix_backup_borg_container_image_self_build: false -matrix_backup_borg_docker_repo: "https://gitlab.com/etke.cc/borgmatic" -matrix_backup_borg_docker_repo_version: main -matrix_backup_borg_docker_src_files_path: "{{ matrix_backup_borg_base_path }}/docker-src" - -# image and postgres version determined automatically, based on detected postgres server version (if enabled), otherwise latest is used -matrix_backup_borg_version: "" -matrix_backup_borg_postgres_version: "" -matrix_backup_borg_borg_version: 1.2.3 -matrix_backup_borg_borgmatic_version: 1.7.6 -matrix_backup_borg_docker_image: "{{ matrix_backup_borg_docker_image_name_prefix }}etke.cc/borgmatic:{{ matrix_backup_borg_version }}" -matrix_backup_borg_docker_image_name_prefix: "{{ 'localhost/' if matrix_backup_borg_container_image_self_build else 'registry.gitlab.com/' }}" -matrix_backup_borg_docker_image_force_pull: "{{ matrix_backup_borg_docker_image.endswith(':latest') or matrix_backup_borg_version | default('') == '' }}" - -# A list of extra arguments to pass to the container -matrix_backup_borg_container_extra_arguments: [] - -# List of systemd services that matrix-backup-borg.service depends on -matrix_backup_borg_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-backup-borg.service wants -matrix_backup_borg_systemd_wanted_services_list: [] - -# systemd calendar configuration for the backup job -# the actual job may run with a delay (see matrix_backup_borg_schedule_randomized_delay_sec) -matrix_backup_borg_schedule: "*-*-* 04:00:00" -# the delay with which the systemd timer may run in relation to the `matrix_backup_borg_schedule` schedule -matrix_backup_borg_schedule_randomized_delay_sec: 2h - -# what directories should be added to backup -matrix_backup_borg_location_source_directories: [] - -# postgres db backup -matrix_backup_borg_postgresql_enabled: true -matrix_backup_borg_supported_postgres_versions: ['12', '13', '14', '15'] -matrix_backup_borg_postgresql_databases: [] -matrix_backup_borg_postgresql_databases_hostname: '' -matrix_backup_borg_postgresql_databases_username: "matrix" -matrix_backup_borg_postgresql_databases_password: "" -matrix_backup_borg_postgresql_databases_port: 5432 - -# target repositories -matrix_backup_borg_location_repositories: [] - -# exclude following paths: -matrix_backup_borg_location_exclude_patterns: [] - -# borg encryption mode, only "repokey-*" and "none" are supported -matrix_backup_borg_encryption: repokey-blake2 - -# private ssh key used to connect to the borg repo -matrix_backup_borg_ssh_key_private: "" - -# allow unencrypted repo access -matrix_backup_borg_unknown_unencrypted_repo_access_is_ok: "{{ matrix_backup_borg_encryption == 'none' }}" - -# borg ssh command with ssh key -matrix_backup_borg_storage_ssh_command: ssh -o "StrictHostKeyChecking accept-new" -i /etc/borgmatic.d/sshkey - -# compression algorithm -matrix_backup_borg_storage_compression: lz4 - -# archive name format -matrix_backup_borg_storage_archive_name_format: matrix-{now:%Y-%m-%d-%H%M%S} - -# repository passphrase -matrix_backup_borg_storage_encryption_passphrase: "" - -# retention configuration -matrix_backup_borg_retention_keep_hourly: 0 -matrix_backup_borg_retention_keep_daily: 7 -matrix_backup_borg_retention_keep_weekly: 4 -matrix_backup_borg_retention_keep_monthly: 12 -matrix_backup_borg_retention_keep_yearly: 2 - -# retention prefix -matrix_backup_borg_retention_prefix: matrix- - -# Default borgmatic configuration template which covers the generic use case. -# You can customize it by controlling the various variables inside it. -# -# For a more advanced customization, you can extend the default (see `matrix_backup_borg_configuration_extension_yaml`) -# or completely replace this variable with your own template. -matrix_backup_borg_configuration_yaml: "{{ lookup('template', 'templates/config.yaml.j2') }}" - -matrix_backup_borg_configuration_extension_yaml: | - # Your custom YAML configuration for borgmatic goes here. - # This configuration extends the default starting configuration (`matrix_borg_configuration_yaml`). - # - # You can override individual variables from the default configuration, or introduce new ones. - # - # If you need something more special, you can take full control by - # completely redefining `matrix_backup_borg_configuration_yaml`. - -matrix_backup_borg_configuration_extension: "{{ matrix_backup_borg_configuration_extension_yaml | from_yaml if matrix_backup_borg_configuration_extension_yaml | from_yaml is mapping else {} }}" - -# Holds the final borgmatic configuration (a combination of the default and its extension). -# You most likely don't need to touch this variable. Instead, see `matrix_backup_borg_configuration_yaml`. -matrix_backup_borg_configuration: "{{ matrix_backup_borg_configuration_yaml | from_yaml | combine(matrix_backup_borg_configuration_extension, recursive=True) }}" diff --git a/roles/custom/matrix-backup-borg/tasks/main.yml b/roles/custom/matrix-backup-borg/tasks/main.yml deleted file mode 100644 index 88d120c0..00000000 --- a/roles/custom/matrix-backup-borg/tasks/main.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: "run_setup | bool and matrix_backup_borg_enabled | bool" - - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - when: "run_setup | bool and matrix_backup_borg_enabled | bool" - tags: - - setup-all - - setup-backup-borg - - install-all - - install-backup-borg - -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - when: "run_setup | bool and not matrix_backup_borg_enabled | bool" - tags: - - setup-all - - setup-backup-borg diff --git a/roles/custom/matrix-backup-borg/tasks/setup_install.yml b/roles/custom/matrix-backup-borg/tasks/setup_install.yml deleted file mode 100644 index 41d95371..00000000 --- a/roles/custom/matrix-backup-borg/tasks/setup_install.yml +++ /dev/null @@ -1,107 +0,0 @@ ---- - -- when: matrix_backup_borg_postgresql_enabled | bool and matrix_backup_borg_version == '' - block: - - name: Fail with matrix_backup_borg_version advice if Postgres not enabled - ansible.builtin.fail: - msg: >- - You are not running a built-in Postgres server (`devture_postgres_enabled: false`), so auto-detecting its version and setting `matrix_backup_borg_version` automatically based on that cannot happen. - Consider setting `matrix_backup_borg_version` to your Postgres version manually. - when: not devture_postgres_enabled - - - ansible.builtin.include_role: - name: galaxy/com.devture.ansible.role.postgres - tasks_from: detect_existing_postgres_version - - - name: Fail if detected Postgres version is unsupported - ansible.builtin.fail: - msg: "You cannot use borg backup with such an old version ({{ devture_postgres_detected_version }}) of Postgres. Consider upgrading - link to docs for upgrading Postgres: docs/maintenance-postgres.md#upgrading-postgresql" - when: "devture_postgres_detected_version not in matrix_backup_borg_supported_postgres_versions" - - - name: Set the correct borg backup version to use - ansible.builtin.set_fact: - matrix_backup_borg_version: "{{ devture_postgres_detected_version }}-{{ matrix_backup_borg_borg_version }}-{{ matrix_backup_borg_borgmatic_version }}" - -- name: Ensure borg paths exist - ansible.builtin.file: - path: "{{ item.path }}" - state: directory - mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - with_items: - - {path: "{{ matrix_backup_borg_config_path }}", when: true} - - {path: "{{ matrix_backup_borg_docker_src_files_path }}", when: true} - when: "item.when | bool" - -- name: Ensure borgmatic config is created - ansible.builtin.copy: - content: "{{ matrix_backup_borg_configuration | to_nice_yaml(indent=2, width=999999) }}" - dest: "{{ matrix_backup_borg_config_path }}/config.yaml" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - mode: 0640 - -- name: Ensure borg passwd is created - ansible.builtin.template: - src: "{{ role_path }}/templates/passwd.j2" - dest: "{{ matrix_backup_borg_config_path }}/passwd" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - mode: 0640 - -- name: Ensure borg ssh key is created - ansible.builtin.template: - src: "{{ role_path }}/templates/sshkey.j2" - dest: "{{ matrix_backup_borg_config_path }}/sshkey" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - mode: 0600 - -- name: Ensure borg image is pulled - community.docker.docker_image: - name: "{{ matrix_backup_borg_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_backup_borg_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_backup_borg_docker_image_force_pull }}" - when: "not matrix_backup_borg_container_image_self_build | bool" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure borg repository is present on self-build - ansible.builtin.git: - repo: "{{ matrix_backup_borg_docker_repo }}" - version: "{{ matrix_backup_borg_docker_repo_version }}" - dest: "{{ matrix_backup_borg_docker_src_files_path }}" - force: "yes" - become: true - become_user: "{{ matrix_user_username }}" - register: matrix_backup_borg_git_pull_results - when: "matrix_backup_borg_container_image_self_build | bool" - -- name: Ensure borg image is built - community.docker.docker_image: - name: "{{ matrix_backup_borg_docker_image }}" - source: build - force_source: "{{ matrix_backup_borg_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" - build: - dockerfile: Dockerfile - path: "{{ matrix_backup_borg_docker_src_files_path }}" - pull: true - when: "matrix_backup_borg_container_image_self_build | bool" - -- name: Ensure matrix-backup-borg.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-backup-borg.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-backup-borg.service" - mode: 0644 - register: matrix_backup_borg_systemd_service_result - -- name: Ensure matrix-backup-borg.timer installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-backup-borg.timer.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-backup-borg.timer" - mode: 0644 diff --git a/roles/custom/matrix-backup-borg/tasks/setup_uninstall.yml b/roles/custom/matrix-backup-borg/tasks/setup_uninstall.yml deleted file mode 100644 index 3d8b08ef..00000000 --- a/roles/custom/matrix-backup-borg/tasks/setup_uninstall.yml +++ /dev/null @@ -1,25 +0,0 @@ ---- - -- name: Check existence of matrix-backup-borg service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-backup-borg.service" - register: matrix_backup_borg_service_stat - -- when: matrix_backup_borg_service_stat.stat.exists | bool - block: - - name: Ensure matrix-backup-borg is stopped - ansible.builtin.service: - name: matrix-backup-borg - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-backup-borg.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-backup-borg.service" - state: absent - - - name: Ensure matrix-backup-borg.timer doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-backup-borg.timer" - state: absent diff --git a/roles/custom/matrix-backup-borg/tasks/validate_config.yml b/roles/custom/matrix-backup-borg/tasks/validate_config.yml deleted file mode 100644 index bb6cd415..00000000 --- a/roles/custom/matrix-backup-borg/tasks/validate_config.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -- name: Fail if required backup-borg settings not defined - ansible.builtin.fail: - msg: >- - You need to define a required configuration setting (`{{ item.name }}`). - when: "item.when | bool and vars[item.name] == ''" - with_items: - - {'name': 'matrix_backup_borg_ssh_key_private', when: true} - - {'name': 'matrix_backup_borg_location_repositories', when: true} - - {'name': 'matrix_backup_borg_postgresql_databases_hostname', when: "{{ matrix_backup_borg_postgresql_enabled }}"} - -- name: Fail if encryption passphrase is undefined unless repository is unencrypted - ansible.builtin.fail: - msg: >- - You need to define a required passphrase using the `matrix_backup_borg_storage_encryption_passphrase` variable. - when: "matrix_backup_borg_storage_encryption_passphrase == '' and matrix_backup_borg_encryption != 'none'" diff --git a/roles/custom/matrix-backup-borg/templates/config.yaml.j2 b/roles/custom/matrix-backup-borg/templates/config.yaml.j2 deleted file mode 100644 index 105c0ce1..00000000 --- a/roles/custom/matrix-backup-borg/templates/config.yaml.j2 +++ /dev/null @@ -1,43 +0,0 @@ -#jinja2: lstrip_blocks: "True", trim_blocks: "True" - -location: - source_directories: {{ matrix_backup_borg_location_source_directories|to_json }} - repositories: {{ matrix_backup_borg_location_repositories|to_json }} - one_file_system: true - exclude_patterns: {{ matrix_backup_borg_location_exclude_patterns|to_json }} - -storage: - compression: {{ matrix_backup_borg_storage_compression|to_json }} - ssh_command: {{ matrix_backup_borg_storage_ssh_command|to_json }} - archive_name_format: {{ matrix_backup_borg_storage_archive_name_format|to_json }} - encryption_passphrase: {{ matrix_backup_borg_storage_encryption_passphrase|to_json }} - unknown_unencrypted_repo_access_is_ok: {{ matrix_backup_borg_unknown_unencrypted_repo_access_is_ok|to_json }} - -retention: - keep_hourly: {{ matrix_backup_borg_retention_keep_hourly|to_json }} - keep_daily: {{ matrix_backup_borg_retention_keep_daily|to_json }} - keep_weekly: {{ matrix_backup_borg_retention_keep_weekly|to_json }} - keep_monthly: {{ matrix_backup_borg_retention_keep_monthly|to_json }} - keep_yearly: {{ matrix_backup_borg_retention_keep_yearly|to_json }} - prefix: {{ matrix_backup_borg_retention_prefix|to_json }} - -consistency: - checks: - - repository - - archives - -hooks: -{% if matrix_backup_borg_postgresql_enabled and matrix_backup_borg_postgresql_databases|length > 0 %} - postgresql_databases: - {% for database in matrix_backup_borg_postgresql_databases %} - - name: {{ database|to_json }} - hostname: {{ matrix_backup_borg_postgresql_databases_hostname|to_json }} - username: {{ matrix_backup_borg_postgresql_databases_username|to_json }} - password: {{ matrix_backup_borg_postgresql_databases_password|to_json }} - port: {{ matrix_backup_borg_postgresql_databases_port | int | to_json }} - {% endfor %} -{% endif %} - after_backup: - - echo "Backup created." - on_error: - - echo "Error while creating a backup." diff --git a/roles/custom/matrix-backup-borg/templates/passwd.j2 b/roles/custom/matrix-backup-borg/templates/passwd.j2 deleted file mode 100644 index d3665cf4..00000000 --- a/roles/custom/matrix-backup-borg/templates/passwd.j2 +++ /dev/null @@ -1,29 +0,0 @@ -{# the passwd file with correct username, UID and GID is mandatory to work with borg over ssh, otherwise ssh connections will fail #} -root:x:0:0:root:/root:/bin/ash -bin:x:1:1:bin:/bin:/sbin/nologin -daemon:x:2:2:daemon:/sbin:/sbin/nologin -adm:x:3:4:adm:/var/adm:/sbin/nologin -lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin -sync:x:5:0:sync:/sbin:/bin/sync -shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown -halt:x:7:0:halt:/sbin:/sbin/halt -mail:x:8:12:mail:/var/mail:/sbin/nologin -news:x:9:13:news:/usr/lib/news:/sbin/nologin -uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin -operator:x:11:0:operator:/root:/sbin/nologin -man:x:13:15:man:/usr/man:/sbin/nologin -postmaster:x:14:12:postmaster:/var/mail:/sbin/nologin -cron:x:16:16:cron:/var/spool/cron:/sbin/nologin -ftp:x:21:21::/var/lib/ftp:/sbin/nologin -sshd:x:22:22:sshd:/dev/null:/sbin/nologin -at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin -squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin -xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin -games:x:35:35:games:/usr/games:/sbin/nologin -cyrus:x:85:12::/usr/cyrus:/sbin/nologin -vpopmail:x:89:89::/var/vpopmail:/sbin/nologin -ntp:x:123:123:NTP:/var/empty:/sbin/nologin -smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin -guest:x:405:100:guest:/dev/null:/sbin/nologin -{{ matrix_user_username }}:x:{{ matrix_user_uid }}:{{ matrix_user_gid }}:Matrix:/tmp:/bin/ash -nobody:x:65534:65534:nobody:/:/sbin/nologin diff --git a/roles/custom/matrix-backup-borg/templates/sshkey.j2 b/roles/custom/matrix-backup-borg/templates/sshkey.j2 deleted file mode 100644 index 999cf38d..00000000 --- a/roles/custom/matrix-backup-borg/templates/sshkey.j2 +++ /dev/null @@ -1 +0,0 @@ -{{ matrix_backup_borg_ssh_key_private }} diff --git a/roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.service.j2 b/roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.service.j2 deleted file mode 100644 index 900369b9..00000000 --- a/roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.service.j2 +++ /dev/null @@ -1,65 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=Matrix Borg Backup -{% for service in matrix_backup_borg_systemd_required_services_list %} -Requires={{ service }} -After={{ service }} -{% endfor %} -{% for service in matrix_backup_borg_systemd_wanted_services_list %} -Wants={{ service }} -{% endfor %} -DefaultDependencies=no - -[Service] -Type=oneshot -Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-backup-borg 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-backup-borg 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-backup-borg \ - --log-driver=none \ - --cap-drop=ALL \ - --read-only \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --network={{ matrix_docker_network }} \ - --tmpfs=/tmp:rw,noexec,nosuid,size=100m \ - --mount type=bind,src={{ matrix_backup_borg_config_path }}/passwd,dst=/etc/passwd,ro \ - --mount type=bind,src={{ matrix_backup_borg_config_path }},dst=/etc/borgmatic.d,ro \ - {% for source in matrix_backup_borg_location_source_directories %} - --mount type=bind,src={{ source }},dst={{ source }},ro \ - {% endfor %} - {% for arg in matrix_backup_borg_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - {{ matrix_backup_borg_docker_image }} \ - sh -c "borgmatic rcreate --encryption {{ matrix_backup_borg_encryption }}" - -# The `CAP_DAC_OVERRIDE` capability is required, so that `root` in the container -# can read the `/etc/borgmatic.d/config.yaml` (`{{ matrix_backup_borg_config_path }}/config.yaml`) file, -# owned by `matrix:matrix` on the filesystem. -# -# `/root` is mountes as temporary filesystem, because we're using `--read-only` and because -# Borgmatic tries to write to at least a few paths under `/root` (`.config`, `.ssh`, `.borgmatic`). -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-backup-borg \ - --log-driver=none \ - --cap-drop=ALL \ - --cap-add=CAP_DAC_OVERRIDE \ - --read-only \ - --network={{ matrix_docker_network }} \ - --tmpfs=/root:rw,noexec,nosuid,size=100m \ - --tmpfs=/tmp:rw,noexec,nosuid,size=100m \ - --mount type=bind,src={{ matrix_backup_borg_config_path }}/passwd,dst=/etc/passwd,ro \ - --mount type=bind,src={{ matrix_backup_borg_config_path }},dst=/etc/borgmatic.d,ro \ - {% for source in matrix_backup_borg_location_source_directories %} - --mount type=bind,src={{ source }},dst={{ source }},ro \ - {% endfor %} - {% for arg in matrix_backup_borg_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - {{ matrix_backup_borg_docker_image }} - -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-backup-borg 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-backup-borg 2>/dev/null || true' -SyslogIdentifier=matrix-backup-borg - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.timer.j2 b/roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.timer.j2 deleted file mode 100644 index fdafef76..00000000 --- a/roles/custom/matrix-backup-borg/templates/systemd/matrix-backup-borg.timer.j2 +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=Matrix Borg Backup timer - -[Timer] -Unit=matrix-backup-borg.service -OnCalendar={{ matrix_backup_borg_schedule }} -RandomizedDelaySec={{ matrix_backup_borg_schedule_randomized_delay_sec }} - -[Install] -WantedBy=timers.target diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml index 496c4c75..109ecd72 100644 --- a/roles/custom/matrix_playbook_migration/defaults/main.yml +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -5,3 +5,6 @@ matrix_playbook_migration_matrix_prometheus_node_exporter_migration_validation_e # Controls if (`matrix_prometheus_postgres_exporter` -> `prometheus_postgres_exporter`) validation will run. matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_validation_enabled: true + +# Controls if (`matrix_backup_borg` -> `backup_borg`) validation will run. +matrix_playbook_migration_matrix_backup_borg_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index b04c9e0e..4f3d7931 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -80,3 +80,20 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_prometheus_postgres_exporter_` -> `prometheus_postgres_exporter_`). We found usage of the following variables: {{ matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_backup_borg_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_backup_borg_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_backup_borg_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_backup_borg variables + ansible.builtin.fail: + msg: >- + The matrix-backup-borg role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/backup_borg. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_backup_borg_` -> `backup_borg_`). + We found usage of the following variables: {{ matrix_playbook_migration_backup_borg_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_backup_borg_migration_vars | length > 0" + + From 38904c08b05148ca2fb3d0d3911058835c1868df Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 11:01:54 +0200 Subject: [PATCH 126/382] Wire backup_borg_username It's probably unnecessary, as this user is only used in the borg container internally, but.. It doesn't hurt to set it to `matrix`. --- group_vars/matrix_servers | 1 + .../custom/matrix_playbook_migration/tasks/validate_config.yml | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index fbae0304..734364db 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1882,6 +1882,7 @@ backup_borg_enabled: false backup_borg_identifier: matrix-backup-borg +backup_borg_username: "{{ matrix_user_username }}" backup_borg_uid: "{{ matrix_user_uid }}" backup_borg_gid: "{{ matrix_user_gid }}" diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 4f3d7931..f588ed08 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -95,5 +95,3 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_backup_borg_` -> `backup_borg_`). We found usage of the following variables: {{ matrix_playbook_migration_backup_borg_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_backup_borg_migration_vars | length > 0" - - From 23f7720247628e6c9f0e9a587a4d727c403be46d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 11:44:19 +0200 Subject: [PATCH 127/382] Add missing backup_borg_base_path override --- group_vars/matrix_servers | 2 ++ 1 file changed, 2 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 734364db..c8c4bd61 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1882,6 +1882,8 @@ backup_borg_enabled: false backup_borg_identifier: matrix-backup-borg +backup_borg_base_path: "{{ matrix_base_data_path }}/backup-borg" + backup_borg_username: "{{ matrix_user_username }}" backup_borg_uid: "{{ matrix_user_uid }}" backup_borg_gid: "{{ matrix_user_gid }}" From 65730b84d35c75dffab374079fe5c67b4619e66e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 11:51:11 +0200 Subject: [PATCH 128/382] Upgrade backup_borg (v1.2.3-1.7.5-0 -> v1.2.3-1.7.5-1) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 75449124..fa5290ff 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git - version: v1.2.3-1.7.5-0 + version: v1.2.3-1.7.5-1 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: fb09fd26f877372417d5586f1e79e83f983f0bd6 From 266195ab450421f2376974483d1bc68c1e62b7d5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 12:25:05 +0200 Subject: [PATCH 129/382] Upgrade backup_borg (v1.2.3-1.7.5-1 -> v1.2.3-1.7.6-0) Supersedes https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2472 --- group_vars/matrix_servers | 2 ++ requirements.yml | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index c8c4bd61..b1d0ffcc 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1890,6 +1890,8 @@ backup_borg_gid: "{{ matrix_user_gid }}" backup_borg_container_network: "{{ devture_postgres_container_network if devture_postgres_enabled else backup_borg_identifier }}" +backup_borg_postgresql_version_detection_devture_postgres_role_name: "{{ 'galaxy/com.devture.ansible.role.postgres' if devture_postgres_enabled else '' }}" + backup_borg_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm32', 'arm64'] }}" backup_borg_postgresql_enabled: "{{ devture_postgres_enabled }}" diff --git a/requirements.yml b/requirements.yml index fa5290ff..1244c465 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git - version: v1.2.3-1.7.5-1 + version: v1.2.3-1.7.6-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: fb09fd26f877372417d5586f1e79e83f983f0bd6 From 33b4f7031b24a58613bcefc90d045652b0f0799f Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Mon, 13 Feb 2023 10:44:42 +0000 Subject: [PATCH 130/382] restore borg prefixes --- group_vars/matrix_servers | 2 ++ 1 file changed, 2 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index b1d0ffcc..f5fab4e6 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1881,6 +1881,8 @@ matrix_bot_draupnir_systemd_required_services_list: | backup_borg_enabled: false backup_borg_identifier: matrix-backup-borg +backup_borg_retention_prefix: matrix- +backup_borg_storage_archive_name_format: matrix-{now:%Y-%m-%d-%H%M%S} backup_borg_base_path: "{{ matrix_base_data_path }}/backup-borg" From 5d146219167754a05ea58bc90068ade537c11bef Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 13:08:53 +0200 Subject: [PATCH 131/382] Fix outdated documentation leading to matrix-backup-borg --- docs/configuring-playbook-backup-borg.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-backup-borg.md b/docs/configuring-playbook-backup-borg.md index 2f6ab5ad..6c9e7f80 100644 --- a/docs/configuring-playbook-backup-borg.md +++ b/docs/configuring-playbook-backup-borg.md @@ -64,7 +64,7 @@ To backup without encryption, add `backup_borg_encryption: 'none'` to your vars. `backup_borg_location_source_directories` defines the list of directories to back up: it's set to `{{ matrix_base_data_path }}` by default, which is the base directory for every service's data, such as Synapse, Postgres and the bridges. You might want to exclude certain directories or file patterns from the backup using the `backup_borg_location_exclude_patterns` variable. -Check the `roles/custom/matrix-backup-borg/defaults/main.yml` file for the full list of available options. +Check the [backup_borg role](https://gitlab.com/etke.cc/roles/backup_borg)'s [defaults/main.yml](https://gitlab.com/etke.cc/roles/backup_borg/-/blob/main/defaults/main.yml) file for the full list of available options. ## Installing From 06f5be4fa549713ef88e830d371f9806b894f9e1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 13:15:01 +0200 Subject: [PATCH 132/382] Announce matrix-backup-borg extraction --- CHANGELOG.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3f74e70..99acdcb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,14 @@ +# 2023-02-13 + +## The matrix-backup-borg role lives independently now + +**TLDR**: the `matrix-backup-borg` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +Thanks to [moan0s](https://github.com/moan0s), the `matrix-backup-borg` role (which configures [Borg backups](docs/configuring-playbook-backup-borg.md)) has been extracted from the playbook and now lives in its [own repository](https://gitlab.com/etke.cc/roles/borg_backup). This makes it possible to easily use it in other Ansible playbooks and will become part of [nextcloud-docker-ansible-deploy](https://github.com/spantaleev/nextcloud-docker-ansible-deploy) soon. + +You need to **update you roles** (`just roles` or `make roles`) regardless of whether you're enabling Borg backup functionality or not. If you're making use of Borg backups via this playbook, you will need to update variable references in your `vars.yml` file (`matrix_backup_borg_` -> `backup_borg_`). + + # 2023-02-12 ## Reverse-proxy configuration changes and initial Traefik support @@ -128,7 +139,7 @@ Additional details are available in [Setting up Draupnir](docs/configuring-playb **TLDR**: the `matrix-prometheus-postgres-exporter` role is now included from another repository. Some variables have been renamed. All functionality remains intact. -The `matrix-prometheus-postgres-exporter` role (which configures [Prometheus Postgres Exporter](https://github.com/prometheus-community/postgres_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter. +The `matrix-prometheus-postgres-exporter` role (which configures [Prometheus Postgres Exporter](https://github.com/prometheus-community/postgres_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter It's still part of the playbook, but is now installed via `ansible-galaxy` (by running `just roles` / `make roles`). Some variables have been renamed (`matrix_prometheus_postgres_exporter_` -> `prometheus_postgres_exporter_`, etc.). The playbook will report all variables that you need to rename to get upgraded. All functionality remains intact. @@ -172,7 +183,7 @@ We've also added `no-multicast-peers` to the default Coturn configuration, but w **TLDR**: the `matrix-prometheus-node-exporter` role is now included from another repository. Some variables have been renamed. All functionality remains intact. -The `matrix-prometheus-node-exporter` role (which configures [Prometheus node exporter](https://github.com/prometheus/node_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_node_exporter. +The `matrix-prometheus-node-exporter` role (which configures [Prometheus node exporter](https://github.com/prometheus/node_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_node_exporter It's still part of the playbook, but is now installed via `ansible-galaxy` (by running `just roles` / `make roles`). Some variables have been renamed (`matrix_prometheus_node_exporter_` -> `prometheus_node_exporter_`, etc.). The playbook will report all variables that you need to rename to get upgraded. All functionality remains intact. From 80935a5194738818eec2a816d2f626914e07dd55 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 14:14:25 +0200 Subject: [PATCH 133/382] Fix broken backup_borg link Referenced here: https://github.com/spantaleev/matrix-docker-ansible-deploy/commit/06f5be4fa549713ef88e830d371f9806b894f9e1#r100414988 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 99acdcb9..3b2716ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ **TLDR**: the `matrix-backup-borg` role is now included from another repository. Some variables have been renamed. All functionality remains intact. -Thanks to [moan0s](https://github.com/moan0s), the `matrix-backup-borg` role (which configures [Borg backups](docs/configuring-playbook-backup-borg.md)) has been extracted from the playbook and now lives in its [own repository](https://gitlab.com/etke.cc/roles/borg_backup). This makes it possible to easily use it in other Ansible playbooks and will become part of [nextcloud-docker-ansible-deploy](https://github.com/spantaleev/nextcloud-docker-ansible-deploy) soon. +Thanks to [moan0s](https://github.com/moan0s), the `matrix-backup-borg` role (which configures [Borg backups](docs/configuring-playbook-backup-borg.md)) has been extracted from the playbook and now lives in its [own repository](https://gitlab.com/etke.cc/roles/backup_borg). This makes it possible to easily use it in other Ansible playbooks and will become part of [nextcloud-docker-ansible-deploy](https://github.com/spantaleev/nextcloud-docker-ansible-deploy) soon. You need to **update you roles** (`just roles` or `make roles`) regardless of whether you're enabling Borg backup functionality or not. If you're making use of Borg backups via this playbook, you will need to update variable references in your `vars.yml` file (`matrix_backup_borg_` -> `backup_borg_`). From f6f7bbd2a1fdc701ff2affe92f289882e1965c3f Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Mon, 13 Feb 2023 12:54:55 +0000 Subject: [PATCH 134/382] Update coturn 4.6.1-r1 -> 4.6.1-r2 --- roles/custom/matrix-coturn/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 78efc73c..47577a46 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -8,7 +8,7 @@ matrix_coturn_container_image_self_build_repo: "https://github.com/coturn/coturn matrix_coturn_container_image_self_build_repo_version: "docker/{{ matrix_coturn_version }}" matrix_coturn_container_image_self_build_repo_dockerfile_path: "docker/coturn/alpine/Dockerfile" -matrix_coturn_version: 4.6.1-r1 +matrix_coturn_version: 4.6.1-r2 matrix_coturn_docker_image: "{{ matrix_coturn_docker_image_name_prefix }}coturn/coturn:{{ matrix_coturn_version }}-alpine" matrix_coturn_docker_image_name_prefix: "{{ 'localhost/' if matrix_coturn_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_coturn_docker_image_force_pull: "{{ matrix_coturn_docker_image.endswith(':latest') }}" From 3d9aa8387e4adc37bbbb997155dfe3a9042241fc Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 15:08:42 +0200 Subject: [PATCH 135/382] Add (native) Traefik support to synapse-admin Previously, it had to go through matrix-nginx-proxy. It's exposed to Traefik directly via container labels now. --- group_vars/matrix_servers | 11 +++-- .../matrix-synapse-admin/defaults/main.yml | 42 ++++++++++++++++++- .../matrix-synapse-admin/tasks/main.yml | 2 +- .../tasks/setup_install.yml | 22 +++++++++- .../tasks/setup_uninstall.yml | 5 +++ .../tasks/validate_config.yml | 21 ++++++++++ .../matrix-synapse-admin/templates/labels.j2 | 25 +++++++++++ .../systemd/matrix-synapse-admin.service.j2 | 14 ++++++- 8 files changed, 133 insertions(+), 9 deletions(-) create mode 100644 roles/custom/matrix-synapse-admin/templates/labels.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index f5fab4e6..3abc1d43 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3159,13 +3159,18 @@ matrix_synapse_reverse_proxy_companion_access_log_syslog_integration_server_port matrix_synapse_admin_enabled: false -# Normally, matrix-nginx-proxy is enabled and nginx can reach Synapse Admin over the container network. -# If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose -# Synapse Admin's HTTP port to the local host. +matrix_synapse_admin_nginx_proxy_integration_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" + matrix_synapse_admin_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8766') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" +matrix_synapse_admin_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + +matrix_synapse_admin_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" +matrix_synapse_admin_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_synapse_admin_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" + ###################################################################### # # /matrix-synapse-admin diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 0f5c7af6..b95632a7 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -4,19 +4,57 @@ matrix_synapse_admin_enabled: true +# A path on host where all related files will be saved +matrix_synapse_admin_base_path: "{{ matrix_base_data_path }}/synapse-admin" +matrix_synapse_admin_docker_src_files_path: "{{ matrix_synapse_admin_base_path }}/docker-src" + +# Specifies whether this role will integrate with the matrix-nginx-proxy role +matrix_synapse_admin_nginx_proxy_integration_enabled: false + matrix_synapse_admin_container_image_self_build: false matrix_synapse_admin_container_image_self_build_repo: "https://github.com/Awesome-Technologies/synapse-admin.git" -matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src" - matrix_synapse_admin_version: 0.8.7 matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:{{ matrix_synapse_admin_version }}" matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}" +# The base container network +matrix_synapse_admin_container_network: matrix-synapse-admin + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# +# Use this to expose this container to another reverse proxy, which runs in a different container network, +# without exposing all other container services to that other reverse-proxy. +matrix_synapse_admin_container_additional_networks: [] + # A list of extra arguments to pass to the container matrix_synapse_admin_container_extra_arguments: [] +# matrix_synapse_admin_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_synapse_admin_container_labels_additional_labels`. +matrix_synapse_admin_container_labels_traefik_enabled: true +matrix_synapse_admin_container_labels_traefik_docker_network: "{{ matrix_synapse_admin_container_network }}" +matrix_synapse_admin_container_labels_traefik_hostname: "{{ matrix_server_fqn_matrix }}" +# The path prefix must not end with a slash +matrix_synapse_admin_container_labels_traefik_path_prefix: /synapse-admin +matrix_synapse_admin_container_labels_traefik_rule: "Host(`{{ matrix_synapse_admin_container_labels_traefik_hostname }}`) && PathPrefix(`{{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }}`)" +matrix_synapse_admin_container_labels_traefik_entrypoints: web-secure +matrix_synapse_admin_container_labels_traefik_tls: "{{ matrix_synapse_admin_container_labels_traefik_entrypoints != 'web' }}" +matrix_synapse_admin_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# matrix_synapse_admin_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_synapse_admin_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_synapse_admin_container_labels_additional_labels: '' + # List of systemd services that matrix-synapse-admin.service depends on matrix_synapse_admin_systemd_required_services_list: ['docker.service'] diff --git a/roles/custom/matrix-synapse-admin/tasks/main.yml b/roles/custom/matrix-synapse-admin/tasks/main.yml index 0b1664cf..f38f1d5e 100644 --- a/roles/custom/matrix-synapse-admin/tasks/main.yml +++ b/roles/custom/matrix-synapse-admin/tasks/main.yml @@ -1,7 +1,7 @@ --- - block: - - when: matrix_synapse_admin_enabled | bool + - when: matrix_synapse_admin_enabled | bool and matrix_synapse_admin_nginx_proxy_integration_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" tags: - setup-all diff --git a/roles/custom/matrix-synapse-admin/tasks/setup_install.yml b/roles/custom/matrix-synapse-admin/tasks/setup_install.yml index 57e59148..2eefe06f 100644 --- a/roles/custom/matrix-synapse-admin/tasks/setup_install.yml +++ b/roles/custom/matrix-synapse-admin/tasks/setup_install.yml @@ -1,5 +1,21 @@ --- +- name: Ensure matrix-synapse-admin path exist + ansible.builtin.file: + path: "{{ matrix_synapse_admin_base_path }}" + state: directory + mode: 0700 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-synapse-admin labels file is created + ansible.builtin.template: + src: "{{ role_path }}/templates/labels.j2" + dest: "{{ matrix_synapse_admin_base_path }}/labels" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + mode: 0640 + - name: Ensure matrix-synapse-admin image is pulled community.docker.docker_image: name: "{{ matrix_synapse_admin_docker_image }}" @@ -35,9 +51,13 @@ pull: true when: matrix_synapse_admin_container_image_self_build | bool +- name: Ensure matrix-synapse-admin container network is created + community.general.docker_network: + name: "{{ matrix_synapse_admin_container_network }}" + driver: bridge + - name: Ensure matrix-synapse-admin.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-synapse-admin.service.j2" dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-admin.service" mode: 0644 - register: matrix_synapse_admin_systemd_service_result diff --git a/roles/custom/matrix-synapse-admin/tasks/setup_uninstall.yml b/roles/custom/matrix-synapse-admin/tasks/setup_uninstall.yml index 3828b735..69820a00 100644 --- a/roles/custom/matrix-synapse-admin/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-synapse-admin/tasks/setup_uninstall.yml @@ -18,3 +18,8 @@ ansible.builtin.file: path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-admin.service" state: absent + + - name: Ensure matrix-synapse-admin directory doesn't exist + ansible.builtin.file: + path: "{{ matrix_synapse_admin_base_path }}" + state: absent diff --git a/roles/custom/matrix-synapse-admin/tasks/validate_config.yml b/roles/custom/matrix-synapse-admin/tasks/validate_config.yml index bf5fe69e..41f19b75 100644 --- a/roles/custom/matrix-synapse-admin/tasks/validate_config.yml +++ b/roles/custom/matrix-synapse-admin/tasks/validate_config.yml @@ -10,3 +10,24 @@ - {'old': 'matrix_synapse_admin_docker_repo', 'new': 'matrix_synapse_admin_container_self_build_repo'} - {'old': 'matrix_synapse_admin_container_self_build', 'new': 'matrix_synapse_admin_container_image_self_build'} - {'old': 'matrix_synapse_admin_container_self_build_repo', 'new': 'matrix_synapse_admin_container_image_self_build_repo'} + +- when: matrix_synapse_admin_container_labels_traefik_enabled | bool + block: + - name: Fail if required matrix-synapse-admin Traefik settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_synapse_admin_container_labels_traefik_hostname + - matrix_synapse_admin_container_labels_traefik_path_prefix + + # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). + # Knowing that `matrix_synapse_admin_container_labels_traefik_path_prefix` does not end with a slash + # ensures we know how to set these routes up without having to do "does it end with a slash" checks elsewhere. + - name: Fail if matrix_synapse_admin_container_labels_traefik_path_prefix ends with a slash + ansible.builtin.fail: + msg: >- + matrix_synapse_admin_container_labels_traefik_path_prefix (`{{ matrix_synapse_admin_container_labels_traefik_path_prefix }}`) must not end with a slash. + Example: `/synapse-admin`. + when: "matrix_synapse_admin_container_labels_traefik_path_prefix[-1] == '/'" diff --git a/roles/custom/matrix-synapse-admin/templates/labels.j2 b/roles/custom/matrix-synapse-admin/templates/labels.j2 new file mode 100644 index 00000000..772fdad8 --- /dev/null +++ b/roles/custom/matrix-synapse-admin/templates/labels.j2 @@ -0,0 +1,25 @@ +{% if matrix_synapse_admin_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_synapse_admin_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_synapse_admin_container_labels_traefik_docker_network }} +{% endif %} + +traefik.http.middlewares.matrix-synapse-admin-slashless-redirect.redirectregex.regex=({{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-synapse-admin-slashless-redirect.redirectregex.replacement=${1}/ + +traefik.http.middlewares.matrix-synapse-admin-replacepath.replacepath.path=/ + +traefik.http.routers.matrix-synapse-admin.rule={{ matrix_synapse_admin_container_labels_traefik_rule }} +traefik.http.routers.matrix-synapse-admin.middlewares=matrix-synapse-admin-slashless-redirect,matrix-synapse-admin-replacepath +traefik.http.routers.matrix-synapse-admin.service=matrix-synapse-admin +traefik.http.routers.matrix-synapse-admin.entrypoints={{ matrix_synapse_admin_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-synapse-admin.tls={{ matrix_synapse_admin_container_labels_traefik_tls | to_json }} +{% if matrix_synapse_admin_container_labels_traefik_tls %} +traefik.http.routers.matrix-synapse-admin.tls.certResolver={{ matrix_synapse_admin_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-synapse-admin.loadbalancer.server.port=80 +{% endif %} + +{{ matrix_synapse_admin_container_labels_additional_labels }} diff --git a/roles/custom/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 b/roles/custom/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 index 9bae6e03..ba56374c 100644 --- a/roles/custom/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 +++ b/roles/custom/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 @@ -16,24 +16,34 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-admin 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-admin 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-synapse-admin \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-synapse-admin \ --log-driver=none \ --cap-drop=ALL \ --cap-add=CHOWN \ --cap-add=NET_BIND_SERVICE \ --cap-add=SETUID \ --cap-add=SETGID \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_synapse_admin_container_network }} \ {% if matrix_synapse_admin_container_http_host_bind_port %} -p {{ matrix_synapse_admin_container_http_host_bind_port }}:80 \ {% endif %} + --label-file={{ matrix_synapse_admin_base_path }}/labels \ {% for arg in matrix_synapse_admin_container_extra_arguments %} {{ arg }} \ {% endfor %} {{ matrix_synapse_admin_docker_image }} +{% for network in matrix_synapse_admin_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-synapse-admin +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-synapse-admin + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-admin 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-admin 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-synapse-admin From 6cda711c0b74ccb6ec192cb2a5947d4b27e39d29 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 15:37:12 +0200 Subject: [PATCH 136/382] Fix incorrect host_bind_port syntax (extra `:`) affecting certain deployments Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2474 Seems like this affected all "own webserver" deployments, which required port exposure. `playbook-managed-traefik` and `playbook-managed-nginx` were not affected. --- group_vars/matrix_servers | 64 +++++++++++++++++++-------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 3abc1d43..cabc149a 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -502,7 +502,7 @@ matrix_appservice_webhooks_container_image_self_build: "{{ matrix_architecture ! # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-webhooks over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-webhooks' client-server port to the local host. -matrix_appservice_webhooks_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_appservice_webhooks_matrix_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_appservice_webhooks_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '' ~ matrix_appservice_webhooks_matrix_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_appservice_webhooks_appservice_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'webhook.as.token', rounds=655555) | to_uuid }}" @@ -584,7 +584,7 @@ matrix_appservice_irc_container_image_self_build: "{{ matrix_architecture != 'am # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-irc over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-irc's client-server port to the local host. -matrix_appservice_irc_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9999') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_appservice_irc_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9999') if matrix_playbook_service_host_bind_interface_prefix else '' }}" # The IRC bridge docs say that if homeserver presence is disabled, it's better to also disable # IRC bridge presence, for performance reasons. @@ -840,7 +840,7 @@ matrix_mautrix_facebook_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_facebook_public_endpoint: "/{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'facebook', rounds=655555) | to_uuid }}" -matrix_mautrix_facebook_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9008') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_mautrix_facebook_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9008') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -891,7 +891,7 @@ matrix_mautrix_googlechat_appservice_token: "{{ '%s' | format(matrix_homeserver_ matrix_mautrix_googlechat_homeserver_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'gc.hs.token', rounds=655555) | to_uuid }}" -matrix_mautrix_googlechat_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9007') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_mautrix_googlechat_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9007') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_googlechat_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -933,7 +933,7 @@ matrix_mautrix_hangouts_appservice_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_hangouts_homeserver_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'ho.hs.token', rounds=655555) | to_uuid }}" -matrix_mautrix_hangouts_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9007') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_mautrix_hangouts_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9007') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_hangouts_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -1080,7 +1080,7 @@ matrix_mautrix_telegram_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_telegram_public_endpoint: "/{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'telegram', rounds=655555) | to_uuid }}" -matrix_mautrix_telegram_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9006') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_mautrix_telegram_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9006') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_mautrix_telegram_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" @@ -1350,7 +1350,7 @@ matrix_mx_puppet_twitter_homeserver_token: "{{ '%s' | format(matrix_homeserver_g matrix_mx_puppet_twitter_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" -matrix_mx_puppet_twitter_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_mx_puppet_twitter_appservice_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_mx_puppet_twitter_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '' ~ matrix_mx_puppet_twitter_appservice_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # Postgres is the default, except if not using internal Postgres server matrix_mx_puppet_twitter_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -1610,7 +1610,7 @@ matrix_bot_maubot_registration_shared_secret: |- }[matrix_homeserver_implementation] }} -matrix_bot_maubot_management_interface_http_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_bot_maubot_management_interface_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_bot_maubot_management_interface_http_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '' ~ matrix_bot_maubot_management_interface_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # Postgres is the default, except if not using internal Postgres server matrix_bot_maubot_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -1806,7 +1806,7 @@ matrix_bot_go_neb_systemd_required_services_list: | (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) }} -matrix_bot_go_neb_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':4050') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_bot_go_neb_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '4050') if matrix_playbook_service_host_bind_interface_prefix else '' }}" ###################################################################### # @@ -1970,8 +1970,8 @@ matrix_corporal_container_image_self_build: "{{ matrix_architecture not in ['amd # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-corporal over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-corporal's web-server ports to the local host. -matrix_corporal_container_http_gateway_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':41080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" -matrix_corporal_container_http_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':41081') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_corporal_container_http_gateway_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '41080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_corporal_container_http_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '41081') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_corporal_systemd_required_services_list: | {{ @@ -2096,7 +2096,7 @@ matrix_dimension_container_image_self_build: "{{ matrix_architecture != 'amd64' # Normally, matrix-nginx-proxy is enabled and nginx can reach Dimension over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the Dimension HTTP port to the local host. -matrix_dimension_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8184') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_dimension_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8184') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_dimension_homeserver_federationUrl: "{{ matrix_homeserver_container_federation_url }}" @@ -2130,7 +2130,7 @@ matrix_dimension_database_password: "{{ '%s' | format(matrix_homeserver_generic_ matrix_etherpad_enabled: false -matrix_etherpad_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9001') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_etherpad_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9001') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_etherpad_base_url: "{{ 'https://' + matrix_server_fqn_dimension + matrix_etherpad_public_endpoint if matrix_etherpad_mode == 'dimension' else 'https://' + matrix_server_fqn_etherpad + '/' }}" @@ -2191,11 +2191,11 @@ matrix_jitsi_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach jitsi/web over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the Jitsi HTTP port to the local host. -matrix_jitsi_web_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':13080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_jitsi_web_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '13080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" -matrix_jitsi_jvb_container_colibri_ws_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':13090') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_jitsi_jvb_container_colibri_ws_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '13090') if matrix_playbook_service_host_bind_interface_prefix else '' }}" -matrix_jitsi_prosody_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':5280') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_jitsi_prosody_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '5280') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_jitsi_jibri_xmpp_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'jibri', rounds=655555) | to_uuid }}" matrix_jitsi_jicofo_auth_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'jicofo', rounds=655555) | to_uuid }}" @@ -2285,7 +2285,7 @@ matrix_ma1sd_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" # Normally, matrix-nginx-proxy is enabled and nginx can reach ma1sd over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # ma1sd's web-server port. -matrix_ma1sd_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_ma1sd_container_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_ma1sd_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '' ~ matrix_ma1sd_container_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # We enable Synapse integration via its Postgres database by default. @@ -2880,7 +2880,7 @@ matrix_sygnal_enabled: false # If someone instals Prometheus via the playbook, they most likely wish to monitor Sygnal. matrix_sygnal_metrics_prometheus_enabled: "{{ matrix_prometheus_enabled }}" -matrix_sygnal_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':6000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_sygnal_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '6000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" ###################################################################### # @@ -2896,7 +2896,7 @@ matrix_sygnal_container_http_host_bind_port: "{{ (matrix_playbook_service_host_b matrix_ntfy_enabled: false -matrix_ntfy_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':2586') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_ntfy_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '2586') if matrix_playbook_service_host_bind_interface_prefix else '' }}" ###################################################################### # @@ -2933,7 +2933,7 @@ matrix_client_element_container_image_self_build: "{{ matrix_architecture not in # Normally, matrix-nginx-proxy is enabled and nginx can reach Element over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the Element HTTP port to the local host. -matrix_client_element_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8765') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_client_element_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8765') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_client_element_default_hs_url: "{{ matrix_homeserver_url }}" matrix_client_element_default_is_url: "{{ matrix_identity_server_url }}" @@ -2978,7 +2978,7 @@ matrix_client_hydrogen_container_image_self_build: "{{ matrix_architecture not i # Normally, matrix-nginx-proxy is enabled and nginx can reach Hydrogen over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the HTTP port to the local host. -matrix_client_hydrogen_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8768') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_client_hydrogen_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8768') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_client_hydrogen_default_hs_url: "{{ matrix_homeserver_url }}" @@ -3003,7 +3003,7 @@ matrix_client_cinny_container_image_self_build: "{{ matrix_architecture not in [ # Normally, matrix-nginx-proxy is enabled and nginx can reach Cinny over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # the HTTP port to the local host. -matrix_client_cinny_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_client_cinny_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_client_cinny_default_hs_url: "{{ matrix_homeserver_url }}" @@ -3032,10 +3032,10 @@ matrix_synapse_account_threepid_delegates_msisdn: "{{ 'http://matrix-ma1sd:' + m matrix_synapse_container_federation_api_tls_host_bind_port: "{{ matrix_federation_public_port if (matrix_synapse_federation_enabled and matrix_synapse_tls_federation_listener_enabled) else '' }}" # # For exposing the Synapse Metrics API's port (plain HTTP) to the local host. -matrix_synapse_container_metrics_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9100') if matrix_synapse_metrics_enabled and matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_synapse_container_metrics_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9100') if matrix_synapse_metrics_enabled and matrix_playbook_service_host_bind_interface_prefix else '' }}" # # For exposing the Synapse Manhole port (plain HTTP) to the local host. -matrix_synapse_container_manhole_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':9000') if matrix_synapse_metrics_enabled and matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_synapse_container_manhole_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9000') if matrix_synapse_metrics_enabled and matrix_playbook_service_host_bind_interface_prefix else '' }}" # # For exposing the Synapse worker (and metrics) ports to the local host. matrix_synapse_workers_container_host_bind_address: "{{ matrix_playbook_service_host_bind_interface_prefix[0:-1] if (matrix_synapse_workers_enabled and matrix_playbook_service_host_bind_interface_prefix) else '' }}" @@ -3127,8 +3127,8 @@ matrix_synapse_reverse_proxy_companion_enabled: "{{ matrix_synapse_enabled }}" matrix_synapse_reverse_proxy_companion_client_api_client_max_body_size_mb: "{{ matrix_synapse_max_upload_size_mb }}" -matrix_synapse_reverse_proxy_companion_container_client_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8008') if matrix_playbook_service_host_bind_interface_prefix else '' }}" -matrix_synapse_reverse_proxy_companion_container_federation_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8048') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_synapse_reverse_proxy_companion_container_client_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8008') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_synapse_reverse_proxy_companion_container_federation_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8048') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_synapse_reverse_proxy_companion_synapse_workers_enabled: "{{ matrix_synapse_workers_enabled }}" matrix_synapse_reverse_proxy_companion_synapse_workers_list: "{{ matrix_synapse_workers_enabled_list }}" @@ -3161,7 +3161,7 @@ matrix_synapse_admin_enabled: false matrix_synapse_admin_nginx_proxy_integration_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" -matrix_synapse_admin_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8766') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_synapse_admin_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8766') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" @@ -3276,7 +3276,7 @@ matrix_prometheus_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach Prometheus over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Prometheus' HTTP port to the local host. -matrix_prometheus_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':99090005') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_prometheus_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9090') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_prometheus_scraper_synapse_enabled: "{{ matrix_synapse_enabled and matrix_synapse_metrics_enabled }}" matrix_prometheus_scraper_synapse_targets: ['matrix-synapse:{{ matrix_synapse_metrics_port }}'] @@ -3313,7 +3313,7 @@ matrix_grafana_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach Grafana over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Grafana's HTTP port to the local host. -matrix_grafana_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':3000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_grafana_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '3000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_grafana_dashboard_download_urls: | {{ @@ -3352,7 +3352,7 @@ matrix_registration_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-registration over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-registration's HTTP port to the local host. -matrix_registration_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':8767') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_registration_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8767') if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_registration_riot_instance: "{{ ('https://' + matrix_server_fqn_element) if matrix_client_element_enabled else 'https://riot.im/app/' }}" @@ -3403,7 +3403,7 @@ matrix_dendrite_enabled: "{{ matrix_homeserver_implementation == 'dendrite' }}" # you can expose Dendrite's ports to the host. # # For exposing Dendrite's plain HTTP server to the local host. -matrix_dendrite_container_http_host_bind_address: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ ':' ~ matrix_dendrite_http_bind_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_dendrite_container_http_host_bind_address: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ matrix_dendrite_http_bind_port | string) if matrix_playbook_service_host_bind_interface_prefix else '' }}" # # For exposing Dendrite's HTTPS server to the local host. matrix_dendrite_container_https_host_bind_address: "{{ '' if not matrix_dendrite_https_bind_port or not matrix_playbook_service_host_bind_interface_prefix else (matrix_playbook_service_host_bind_interface_prefix + matrix_dendrite_https_bind_port | string) }}" @@ -3543,7 +3543,7 @@ matrix_user_verification_service_systemd_required_services_list: | # If Jitsi is not managed by this playbook, or you otherwise have a need for it, you can expose # matrix-user-verfification-services's client-server port to port 3003. # By default Matrix-User-Verification-Service binds to port 3000, which collides with grafana, therefore this uses port 3003. -matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') else matrix_playbook_service_host_bind_interface_prefix ~ ':3003' }}" +matrix_user_verification_service_container_http_host_bind_port: "{{ '' if (matrix_jitsi_enabled | bool and matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix') else matrix_playbook_service_host_bind_interface_prefix ~ '3003' }}" # URL exposed in the docker network matrix_user_verification_service_container_url: "http://{{ matrix_user_verification_service_container_name }}:3000" From 9f820a506aab53f3e13552ad07f6c0d969b865ed Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Mon, 13 Feb 2023 14:08:09 +0000 Subject: [PATCH 137/382] Update postmoogle 0.9.12 -> 0.9.13 * live SSL certificates reload on file changes (e.g., on automatic certs renewal) * print all errors when trying connection to an SMTP server --- roles/custom/matrix-bot-postmoogle/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-postmoogle/defaults/main.yml b/roles/custom/matrix-bot-postmoogle/defaults/main.yml index 9f3dad24..b3b12fef 100644 --- a/roles/custom/matrix-bot-postmoogle/defaults/main.yml +++ b/roles/custom/matrix-bot-postmoogle/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_postmoogle_docker_repo: "https://gitlab.com/etke.cc/postmoogle.git" matrix_bot_postmoogle_docker_repo_version: "{{ 'main' if matrix_bot_postmoogle_version == 'latest' else matrix_bot_postmoogle_version }}" matrix_bot_postmoogle_docker_src_files_path: "{{ matrix_base_data_path }}/postmoogle/docker-src" -matrix_bot_postmoogle_version: v0.9.12 +matrix_bot_postmoogle_version: v0.9.13 matrix_bot_postmoogle_docker_image: "{{ matrix_bot_postmoogle_docker_image_name_prefix }}etke.cc/postmoogle:{{ matrix_bot_postmoogle_version }}" matrix_bot_postmoogle_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_postmoogle_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_postmoogle_docker_image_force_pull: "{{ matrix_bot_postmoogle_docker_image.endswith(':latest') }}" From b2d8718233006f1365b604a11ec2cfaf13c625ab Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 16:08:51 +0200 Subject: [PATCH 138/382] Fix synapse-admin reverse-proxying regression for "playbook-managed-nginx" Regression since 3d9aa8387e4adc3 --- group_vars/matrix_servers | 2 ++ 1 file changed, 2 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index cabc149a..a1a5f797 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3165,6 +3165,8 @@ matrix_synapse_admin_container_http_host_bind_port: "{{ (matrix_playbook_service matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" +matrix_synapse_admin_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-synapse-admin' }}" + matrix_synapse_admin_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" matrix_synapse_admin_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" From f2ed5e4b0499a24b8b51e7e524800b1993c26cdb Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 18:28:03 +0200 Subject: [PATCH 139/382] Delete /matrix/nginx-proxy/conf.d/matrix-client-element.conf if matrix_nginx_proxy_proxy_element_enabled not enabled --- roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml b/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml index 31ff68f2..5aa63597 100644 --- a/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml +++ b/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml @@ -261,6 +261,12 @@ state: absent when: "not matrix_nginx_proxy_proxy_riot_compat_redirect_enabled | bool" +- name: Ensure Matrix nginx-proxy configuration for Element domain deleted + ansible.builtin.file: + path: "{{ matrix_nginx_proxy_confd_path }}/matrix-client-element.conf" + state: absent + when: "not matrix_nginx_proxy_proxy_element_enabled | bool" + - name: Ensure Matrix nginx-proxy configuration for Hydrogen domain deleted ansible.builtin.file: path: "{{ matrix_nginx_proxy_confd_path }}/matrix-client-hydrogen.conf" From e51e4eec097d5e6976b8083e3a4600d0b3184e0b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 18:33:06 +0200 Subject: [PATCH 140/382] Add (native) Traefik support to matrix-client-element Previously, it had to go through matrix-nginx-proxy. It's exposed to Traefik directly via container labels now --- group_vars/matrix_servers | 11 +++++-- .../matrix-client-element/defaults/main.yml | 29 +++++++++++++++++++ .../tasks/setup_install.yml | 6 ++++ .../tasks/setup_uninstall.yml | 2 +- .../matrix-client-element/templates/labels.j2 | 19 ++++++++++++ .../systemd/matrix-client-element.service.j2 | 14 +++++++-- .../matrix-nginx-proxy/defaults/main.yml | 5 ---- .../matrix-nginx-proxy/templates/labels.j2 | 12 -------- .../matrix-synapse-admin/defaults/main.yml | 4 +-- 9 files changed, 77 insertions(+), 25 deletions(-) create mode 100644 roles/custom/matrix-client-element/templates/labels.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index a1a5f797..ca046040 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2378,7 +2378,7 @@ matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_admin_api_ matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain: "{{ matrix_server_fqn_element if matrix_client_element_enabled else '' }}" matrix_nginx_proxy_proxy_matrix_enabled: true -matrix_nginx_proxy_proxy_element_enabled: "{{ matrix_client_element_enabled }}" +matrix_nginx_proxy_proxy_element_enabled: "{{ matrix_client_element_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled }}" matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled }}" matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" @@ -2398,7 +2398,6 @@ matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true -matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled: "{{ matrix_client_element_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" @@ -2935,6 +2934,14 @@ matrix_client_element_container_image_self_build: "{{ matrix_architecture not in # the Element HTTP port to the local host. matrix_client_element_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8765') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_client_element_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-client-element' }}" + +matrix_client_element_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + +matrix_client_element_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" +matrix_client_element_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_client_element_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" + matrix_client_element_default_hs_url: "{{ matrix_homeserver_url }}" matrix_client_element_default_is_url: "{{ matrix_identity_server_url }}" diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 85d8d6cf..82097e1c 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -18,11 +18,40 @@ matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_ matrix_client_element_data_path: "{{ matrix_base_data_path }}/client-element" matrix_client_element_docker_src_files_path: "{{ matrix_client_element_data_path }}/docker-src" +# The base container network +matrix_client_element_container_network: matrix-client-element + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to a reverse proxy, which runs in a different container network. +matrix_client_element_container_additional_networks: [] + # Controls whether the matrix-client-element container exposes its HTTP port (tcp/8080 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:8765"), or empty string to not expose. matrix_client_element_container_http_host_bind_port: '' +# matrix_client_element_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_client_element_container_labels_additional_labels`. +matrix_client_element_container_labels_traefik_enabled: true +matrix_client_element_container_labels_traefik_docker_network: "{{ matrix_client_element_container_network }}" +matrix_client_element_container_labels_traefik_hostname: "{{ matrix_server_fqn_element }}" +matrix_client_element_container_labels_traefik_rule: "Host(`{{ matrix_client_element_container_labels_traefik_hostname }}`)" +matrix_client_element_container_labels_traefik_entrypoints: web-secure +matrix_client_element_container_labels_traefik_tls: "{{ matrix_client_element_container_labels_traefik_entrypoints != 'web' }}" +matrix_client_element_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# matrix_client_element_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_client_element_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_client_element_container_labels_additional_labels: '' + # A list of extra arguments to pass to the container matrix_client_element_container_extra_arguments: [] diff --git a/roles/custom/matrix-client-element/tasks/setup_install.yml b/roles/custom/matrix-client-element/tasks/setup_install.yml index 813b4da0..fc0532db 100644 --- a/roles/custom/matrix-client-element/tasks/setup_install.yml +++ b/roles/custom/matrix-client-element/tasks/setup_install.yml @@ -84,6 +84,7 @@ group: "{{ matrix_user_groupname }}" with_items: - {src: "{{ role_path }}/templates/nginx.conf.j2", name: "nginx.conf"} + - {src: "{{ role_path }}/templates/labels.j2", name: "labels"} - {src: "{{ matrix_client_element_page_template_welcome_path }}", name: "welcome.html"} - {src: "{{ matrix_client_element_embedded_pages_home_path }}", name: "home.html"} when: "item.src is not none" @@ -96,6 +97,11 @@ - {src: "{{ matrix_client_element_embedded_pages_home_path }}", name: "home.html"} when: "item.src is none" +- name: Ensure Element container network is created + community.general.docker_network: + name: "{{ matrix_client_element_container_network }}" + driver: bridge + - name: Ensure matrix-client-element.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-client-element.service.j2" diff --git a/roles/custom/matrix-client-element/tasks/setup_uninstall.yml b/roles/custom/matrix-client-element/tasks/setup_uninstall.yml index e1b25caa..e0930cc1 100644 --- a/roles/custom/matrix-client-element/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-client-element/tasks/setup_uninstall.yml @@ -19,7 +19,7 @@ path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-client-element.service" state: absent - - name: Ensure Element paths doesn't exist + - name: Ensure Element path doesn't exist ansible.builtin.file: path: "{{ matrix_client_element_data_path }}" state: absent diff --git a/roles/custom/matrix-client-element/templates/labels.j2 b/roles/custom/matrix-client-element/templates/labels.j2 new file mode 100644 index 00000000..85df036f --- /dev/null +++ b/roles/custom/matrix-client-element/templates/labels.j2 @@ -0,0 +1,19 @@ +{% if matrix_client_element_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_client_element_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_client_element_container_labels_traefik_docker_network }} +{% endif %} + +traefik.http.routers.matrix-client-element.rule={{ matrix_client_element_container_labels_traefik_rule }} +traefik.http.routers.matrix-client-element.service=matrix-client-element +traefik.http.routers.matrix-client-element.entrypoints={{ matrix_client_element_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-client-element.tls={{ matrix_client_element_container_labels_traefik_tls | to_json }} +{% if matrix_client_element_container_labels_traefik_tls %} +traefik.http.routers.matrix-client-element.tls.certResolver={{ matrix_client_element_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-client-element.loadbalancer.server.port=8080 +{% endif %} + +{{ matrix_client_element_container_labels_additional_labels }} diff --git a/roles/custom/matrix-client-element/templates/systemd/matrix-client-element.service.j2 b/roles/custom/matrix-client-element/templates/systemd/matrix-client-element.service.j2 index be96cab7..bbb50989 100644 --- a/roles/custom/matrix-client-element/templates/systemd/matrix-client-element.service.j2 +++ b/roles/custom/matrix-client-element/templates/systemd/matrix-client-element.service.j2 @@ -13,15 +13,18 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-client-element 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-client-element 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-client-element \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-client-element \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_client_element_container_network }} \ {% if matrix_client_element_container_http_host_bind_port %} -p {{ matrix_client_element_container_http_host_bind_port }}:8080 \ {% endif %} + --label-file={{ matrix_client_element_data_path }}/labels \ --tmpfs=/tmp:rw,noexec,nosuid,size=10m \ --mount type=bind,src={{ matrix_client_element_data_path }}/nginx.conf,dst=/etc/nginx/nginx.conf,ro \ --mount type=bind,src={{ matrix_client_element_data_path }}/config.json,dst=/app/config.json,ro \ @@ -38,8 +41,15 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endfor %} {{ matrix_client_element_docker_image }} +{% for network in matrix_client_element_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-client-element +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-client-element + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-client-element 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-client-element 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-client-element diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 4ce7e923..b443cc72 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -61,11 +61,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule: "Host(`{{ matrix_ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" -matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_element_hostname: "{{ matrix_server_fqn_element }}" -matrix_nginx_proxy_container_labels_traefik_proxy_element_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_element_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_element_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_hostname: "{{ matrix_server_fqn_hydrogen }}" matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index b6994617..d629ce00 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -39,18 +39,6 @@ traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints={{ matrix_ {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_element_enabled %} -# Element -traefik.http.routers.matrix-nginx-proxy-element.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_element_rule }} -traefik.http.routers.matrix-nginx-proxy-element.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-element.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_element_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_element_tls %} -traefik.http.routers.matrix-nginx-proxy-element.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-element.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - {% if matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled %} # Hydrogen traefik.http.routers.matrix-nginx-proxy-hydrogen.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_rule }} diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index b95632a7..11b89818 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -24,9 +24,7 @@ matrix_synapse_admin_container_network: matrix-synapse-admin # A list of additional container networks that the container would be connected to. # The role does not create these networks, so make sure they already exist. -# -# Use this to expose this container to another reverse proxy, which runs in a different container network, -# without exposing all other container services to that other reverse-proxy. +# Use this to expose this container to a reverse proxy, which runs in a different container network. matrix_synapse_admin_container_additional_networks: [] # A list of extra arguments to pass to the container From e34174b1b4350b180d0922a38efe1c447da473d5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Feb 2023 19:03:10 +0200 Subject: [PATCH 141/382] Add various security headers to matrix-client-element when behind Traefik --- .../matrix-client-element/defaults/main.yml | 66 +++++++++++++++++++ .../matrix-client-element/templates/labels.j2 | 12 ++++ 2 files changed, 78 insertions(+) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 82097e1c..890bc473 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -43,6 +43,72 @@ matrix_client_element_container_labels_traefik_entrypoints: web-secure matrix_client_element_container_labels_traefik_tls: "{{ matrix_client_element_container_labels_traefik_entrypoints != 'web' }}" matrix_client_element_container_labels_traefik_tls_certResolver: default # noqa var-naming +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_client_element_container_labels_traefik_additional_response_headers_custom` +matrix_client_element_container_labels_traefik_additional_response_headers: "{{ matrix_client_element_container_labels_traefik_additional_response_headers_auto | combine(matrix_client_element_container_labels_traefik_additional_response_headers_custom) }}" +matrix_client_element_container_labels_traefik_additional_response_headers_auto: | + {{ + {} + | combine ({'X-XSS-Protection': matrix_client_element_http_header_xss_protection} if matrix_client_element_http_header_xss_protection else {}) + | combine ({'X-Frame-Options': matrix_client_element_http_header_frame_options} if matrix_client_element_http_header_frame_options else {}) + | combine ({'X-Content-Type-Options': matrix_client_element_http_header_content_type_options} if matrix_client_element_http_header_content_type_options else {}) + | combine ({'Content-Security-Policy': matrix_client_element_http_header_content_security_policy} if matrix_client_element_http_header_content_security_policy else {}) + | combine ({'Permission-Policy': matrix_client_element_http_header_content_permission_policy} if matrix_client_element_http_header_content_permission_policy else {}) + | combine ({'Strict-Transport-Security': matrix_client_element_http_header_strict_transport_security} if matrix_client_element_http_header_strict_transport_security and matrix_client_element_container_labels_traefik_tls else {}) + }} +matrix_client_element_container_labels_traefik_additional_response_headers_custom: {} + +# Specifies the value of the `X-XSS-Protection` header +# Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. +# +# Learn more about it is here: +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection +# - https://portswigger.net/web-security/cross-site-scripting/reflected +matrix_client_element_http_header_xss_protection: "1; mode=block" + +# Specifies the value of the `X-Frame-Options` header which controls whether framing can happen. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options +matrix_client_element_http_header_frame_options: SAMEORIGIN + +# Specifies the value of the `X-Content-Type-Options` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options +matrix_client_element_http_header_content_type_options: nosniff + +# Specifies the value of the `Content-Security-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy +matrix_client_element_http_header_content_security_policy: frame-ancestors 'self' + +# Specifies the value of the `Permission-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy +matrix_client_element_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_client_element_floc_optout_enabled else '' }}" + +# Specifies the value of the `Strict-Transport-Security` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +matrix_client_element_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_client_element_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" + +# Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses +# +# Learn more about what it is here: +# - https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea +# - https://paramdeo.com/blog/opting-your-website-out-of-googles-floc-network +# - https://amifloced.org/ +# +# Of course, a better solution is to just stop using browsers (like Chrome), which participate in such tracking practices. +# See: `matrix_client_element_content_permission_policy` +matrix_client_element_floc_optout_enabled: true + +# Controls if HSTS preloading is enabled +# +# In its strongest and recommended form, the [HSTS policy](https://www.chromium.org/hsts) includes all subdomains, and +# indicates a willingness to be "preloaded" into browsers: +# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload` +# For more information visit: +# - https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +# - https://hstspreload.org/#opt-in +# See: `matrix_client_element_http_header_strict_transport_security` +matrix_client_element_hsts_preload_enabled: false + # matrix_client_element_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. # See `../templates/labels.j2` for details. # diff --git a/roles/custom/matrix-client-element/templates/labels.j2 b/roles/custom/matrix-client-element/templates/labels.j2 index 85df036f..5f5cd751 100644 --- a/roles/custom/matrix-client-element/templates/labels.j2 +++ b/roles/custom/matrix-client-element/templates/labels.j2 @@ -5,8 +5,20 @@ traefik.enable=true traefik.docker.network={{ matrix_client_element_container_labels_traefik_docker_network }} {% endif %} +{% set middlewares = [] %} + +{% if matrix_client_element_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_client_element_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-client-element-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-client-element-add-headers'] %} +{% endif %} + traefik.http.routers.matrix-client-element.rule={{ matrix_client_element_container_labels_traefik_rule }} traefik.http.routers.matrix-client-element.service=matrix-client-element +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-client-element.middlewares={{ middlewares | join(',') }} +{% endif %} traefik.http.routers.matrix-client-element.entrypoints={{ matrix_client_element_container_labels_traefik_entrypoints }} traefik.http.routers.matrix-client-element.tls={{ matrix_client_element_container_labels_traefik_tls | to_json }} {% if matrix_client_element_container_labels_traefik_tls %} From ff1338e0036e4d97fcb7d41d584afdb78449fe5c Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 08:31:26 +0200 Subject: [PATCH 142/382] Add support for hosting Element (on Traefik) at a subpath --- .../matrix-client-element/defaults/main.yml | 4 +++- .../tasks/validate_config.yml | 20 +++++++++++++++++++ .../matrix-client-element/templates/labels.j2 | 11 ++++++++++ 3 files changed, 34 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 890bc473..3ebed056 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -38,7 +38,9 @@ matrix_client_element_container_http_host_bind_port: '' matrix_client_element_container_labels_traefik_enabled: true matrix_client_element_container_labels_traefik_docker_network: "{{ matrix_client_element_container_network }}" matrix_client_element_container_labels_traefik_hostname: "{{ matrix_server_fqn_element }}" -matrix_client_element_container_labels_traefik_rule: "Host(`{{ matrix_client_element_container_labels_traefik_hostname }}`)" +# The path prefix must either be `/` or not end with a slash (e.g. `/element`). +matrix_client_element_container_labels_traefik_path_prefix: / +matrix_client_element_container_labels_traefik_rule: "Host(`{{ matrix_client_element_container_labels_traefik_hostname }}`){% if matrix_client_element_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_client_element_container_labels_traefik_path_prefix }}`){% endif %}" matrix_client_element_container_labels_traefik_entrypoints: web-secure matrix_client_element_container_labels_traefik_tls: "{{ matrix_client_element_container_labels_traefik_entrypoints != 'web' }}" matrix_client_element_container_labels_traefik_tls_certResolver: default # noqa var-naming diff --git a/roles/custom/matrix-client-element/tasks/validate_config.yml b/roles/custom/matrix-client-element/tasks/validate_config.yml index 0e252db8..658542fb 100644 --- a/roles/custom/matrix-client-element/tasks/validate_config.yml +++ b/roles/custom/matrix-client-element/tasks/validate_config.yml @@ -26,3 +26,23 @@ when: "vars | dict2items | selectattr('key', 'match', item.old) | list | items2dict" with_items: - {'old': 'matrix_riot_web_.*', 'new': 'matrix_client_element_.*'} + +- when: matrix_client_element_container_labels_traefik_enabled | bool + block: + - name: Fail if required matrix-client-element Traefik settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_synapse_admin_container_labels_traefik_hostname + - matrix_client_element_container_labels_traefik_path_prefix + + # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). + # Knowing that `matrix_client_element_container_labels_traefik_path_prefix` does not end with a slash + # ensures we know how to set these routes up without having to do "does it end with a slash" checks elsewhere. + - name: Fail if matrix_client_element_container_labels_traefik_path_prefix ends with a slash + ansible.builtin.fail: + msg: >- + matrix_client_element_container_labels_traefik_path_prefix (`{{ matrix_client_element_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/element`). + when: "matrix_client_element_container_labels_traefik_path_prefix != '/' and matrix_client_element_container_labels_traefik_path_prefix[-1] == '/'" diff --git a/roles/custom/matrix-client-element/templates/labels.j2 b/roles/custom/matrix-client-element/templates/labels.j2 index 5f5cd751..33fb28e1 100644 --- a/roles/custom/matrix-client-element/templates/labels.j2 +++ b/roles/custom/matrix-client-element/templates/labels.j2 @@ -7,6 +7,17 @@ traefik.docker.network={{ matrix_client_element_container_labels_traefik_docker_ {% set middlewares = [] %} +{% if matrix_client_element_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-client-element-slashless-redirect.redirectregex.regex=({{ matrix_client_element_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-client-element-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-client-element-slashless-redirect'] %} +{% endif %} + +{% if matrix_client_element_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-client-element-strip-prefix.stripprefix.prefixes={{ matrix_client_element_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-client-element-strip-prefix'] %} +{% endif %} + {% if matrix_client_element_container_labels_traefik_additional_response_headers.keys() | length > 0 %} {% for name, value in matrix_client_element_container_labels_traefik_additional_response_headers.items() %} traefik.http.middlewares.matrix-client-element-add-headers.headers.customresponseheaders.{{ name }}={{ value }} From 5ab5f28d14c442091192c7c0ad3d4a64f64003d6 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 08:42:50 +0200 Subject: [PATCH 143/382] Add support for running synapse-admin (on Traefik) at the root path Previously, we had to run it at a subpath, like `/synapse-admin`. We can now dedicate a whole domain and the `/` path to it, should we wish to do so. --- .../custom/matrix-synapse-admin/defaults/main.yml | 4 ++-- .../matrix-synapse-admin/tasks/validate_config.yml | 5 ++--- .../matrix-synapse-admin/templates/labels.j2 | 14 ++++++++++++-- 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 11b89818..cb64b549 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -37,9 +37,9 @@ matrix_synapse_admin_container_extra_arguments: [] matrix_synapse_admin_container_labels_traefik_enabled: true matrix_synapse_admin_container_labels_traefik_docker_network: "{{ matrix_synapse_admin_container_network }}" matrix_synapse_admin_container_labels_traefik_hostname: "{{ matrix_server_fqn_matrix }}" -# The path prefix must not end with a slash +# The path prefix must either be `/` or not end with a slash (e.g. `/synapse-admin`). matrix_synapse_admin_container_labels_traefik_path_prefix: /synapse-admin -matrix_synapse_admin_container_labels_traefik_rule: "Host(`{{ matrix_synapse_admin_container_labels_traefik_hostname }}`) && PathPrefix(`{{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }}`)" +matrix_synapse_admin_container_labels_traefik_rule: "Host(`{{ matrix_synapse_admin_container_labels_traefik_hostname }}`){% if matrix_synapse_admin_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }}`){% endif %}" matrix_synapse_admin_container_labels_traefik_entrypoints: web-secure matrix_synapse_admin_container_labels_traefik_tls: "{{ matrix_synapse_admin_container_labels_traefik_entrypoints != 'web' }}" matrix_synapse_admin_container_labels_traefik_tls_certResolver: default # noqa var-naming diff --git a/roles/custom/matrix-synapse-admin/tasks/validate_config.yml b/roles/custom/matrix-synapse-admin/tasks/validate_config.yml index 41f19b75..48243555 100644 --- a/roles/custom/matrix-synapse-admin/tasks/validate_config.yml +++ b/roles/custom/matrix-synapse-admin/tasks/validate_config.yml @@ -28,6 +28,5 @@ - name: Fail if matrix_synapse_admin_container_labels_traefik_path_prefix ends with a slash ansible.builtin.fail: msg: >- - matrix_synapse_admin_container_labels_traefik_path_prefix (`{{ matrix_synapse_admin_container_labels_traefik_path_prefix }}`) must not end with a slash. - Example: `/synapse-admin`. - when: "matrix_synapse_admin_container_labels_traefik_path_prefix[-1] == '/'" + matrix_synapse_admin_container_labels_traefik_path_prefix (`{{ matrix_synapse_admin_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/synapse-admin`). + when: "matrix_synapse_admin_container_labels_traefik_path_prefix != '/' and matrix_synapse_admin_container_labels_traefik_path_prefix[-1] == '/'" diff --git a/roles/custom/matrix-synapse-admin/templates/labels.j2 b/roles/custom/matrix-synapse-admin/templates/labels.j2 index 772fdad8..c70892a8 100644 --- a/roles/custom/matrix-synapse-admin/templates/labels.j2 +++ b/roles/custom/matrix-synapse-admin/templates/labels.j2 @@ -5,13 +5,23 @@ traefik.enable=true traefik.docker.network={{ matrix_synapse_admin_container_labels_traefik_docker_network }} {% endif %} +{% set middlewares = [] %} + +{% if matrix_synapse_admin_container_labels_traefik_path_prefix != '/' %} traefik.http.middlewares.matrix-synapse-admin-slashless-redirect.redirectregex.regex=({{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }})$ traefik.http.middlewares.matrix-synapse-admin-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-synapse-admin-slashless-redirect'] %} +{% endif %} -traefik.http.middlewares.matrix-synapse-admin-replacepath.replacepath.path=/ +{% if matrix_synapse_admin_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-synapse-admin-strip-prefix.stripprefix.prefixes={{ matrix_synapse_admin_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-synapse-admin-strip-prefix'] %} +{% endif %} traefik.http.routers.matrix-synapse-admin.rule={{ matrix_synapse_admin_container_labels_traefik_rule }} -traefik.http.routers.matrix-synapse-admin.middlewares=matrix-synapse-admin-slashless-redirect,matrix-synapse-admin-replacepath +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-synapse-admin.middlewares={{ middlewares | join(',') }} +{% endif %} traefik.http.routers.matrix-synapse-admin.service=matrix-synapse-admin traefik.http.routers.matrix-synapse-admin.entrypoints={{ matrix_synapse_admin_container_labels_traefik_entrypoints }} traefik.http.routers.matrix-synapse-admin.tls={{ matrix_synapse_admin_container_labels_traefik_tls | to_json }} From 71597132e0d8efac9456ecd984f0750cc62e7d43 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 08:45:32 +0200 Subject: [PATCH 144/382] Move around some matrix-client-element variables --- .../matrix-client-element/defaults/main.yml | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 3ebed056..500c5b47 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -60,6 +60,21 @@ matrix_client_element_container_labels_traefik_additional_response_headers_auto: }} matrix_client_element_container_labels_traefik_additional_response_headers_custom: {} +# matrix_client_element_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_client_element_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_client_element_container_labels_additional_labels: '' + +# A list of extra arguments to pass to the container +matrix_client_element_container_extra_arguments: [] + +# List of systemd services that matrix-client-element.service depends on +matrix_client_element_systemd_required_services_list: ['docker.service'] + # Specifies the value of the `X-XSS-Protection` header # Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. # @@ -111,21 +126,6 @@ matrix_client_element_floc_optout_enabled: true # See: `matrix_client_element_http_header_strict_transport_security` matrix_client_element_hsts_preload_enabled: false -# matrix_client_element_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. -# See `../templates/labels.j2` for details. -# -# Example: -# matrix_client_element_container_labels_additional_labels: | -# my.label=1 -# another.label="here" -matrix_client_element_container_labels_additional_labels: '' - -# A list of extra arguments to pass to the container -matrix_client_element_container_extra_arguments: [] - -# List of systemd services that matrix-client-element.service depends on -matrix_client_element_systemd_required_services_list: ['docker.service'] - # Element config.json customizations matrix_client_element_default_server_name: "{{ matrix_domain }}" matrix_client_element_default_hs_url: "" From c33ed94352e48be29d200a3ff12992a30c9efc39 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 08:49:00 +0200 Subject: [PATCH 145/382] Add security headers to synapse-admin (on Traefik) We've had it on `matrix-nginx-proxy` before, but our initial support for Traefik did not include any of these security headers. --- .../matrix-synapse-admin/defaults/main.yml | 72 ++++++++++++++++++- .../matrix-synapse-admin/templates/labels.j2 | 7 ++ 2 files changed, 76 insertions(+), 3 deletions(-) diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index cb64b549..f150bd35 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -27,6 +27,11 @@ matrix_synapse_admin_container_network: matrix-synapse-admin # Use this to expose this container to a reverse proxy, which runs in a different container network. matrix_synapse_admin_container_additional_networks: [] +# Controls whether the matrix-synapse-admin container exposes its HTTP port (tcp/80 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:8766"), or empty string to not expose. +matrix_synapse_admin_container_http_host_bind_port: '' + # A list of extra arguments to pass to the container matrix_synapse_admin_container_extra_arguments: [] @@ -44,6 +49,21 @@ matrix_synapse_admin_container_labels_traefik_entrypoints: web-secure matrix_synapse_admin_container_labels_traefik_tls: "{{ matrix_synapse_admin_container_labels_traefik_entrypoints != 'web' }}" matrix_synapse_admin_container_labels_traefik_tls_certResolver: default # noqa var-naming +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_synapse_admin_container_labels_traefik_additional_response_headers_custom` +matrix_synapse_admin_container_labels_traefik_additional_response_headers: "{{ matrix_synapse_admin_container_labels_traefik_additional_response_headers_auto | combine(matrix_synapse_admin_container_labels_traefik_additional_response_headers_custom) }}" +matrix_synapse_admin_container_labels_traefik_additional_response_headers_auto: | + {{ + {} + | combine ({'X-XSS-Protection': matrix_synapse_admin_http_header_xss_protection} if matrix_synapse_admin_http_header_xss_protection else {}) + | combine ({'X-Frame-Options': matrix_synapse_admin_http_header_frame_options} if matrix_synapse_admin_http_header_frame_options else {}) + | combine ({'X-Content-Type-Options': matrix_synapse_admin_http_header_content_type_options} if matrix_synapse_admin_http_header_content_type_options else {}) + | combine ({'Content-Security-Policy': matrix_synapse_admin_http_header_content_security_policy} if matrix_synapse_admin_http_header_content_security_policy else {}) + | combine ({'Permission-Policy': matrix_synapse_admin_http_header_content_permission_policy} if matrix_synapse_admin_http_header_content_permission_policy else {}) + | combine ({'Strict-Transport-Security': matrix_synapse_admin_http_header_strict_transport_security} if matrix_synapse_admin_http_header_strict_transport_security and matrix_synapse_admin_container_labels_traefik_tls else {}) + }} +matrix_synapse_admin_container_labels_traefik_additional_response_headers_custom: {} + # matrix_synapse_admin_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. # See `../templates/labels.j2` for details. # @@ -59,10 +79,56 @@ matrix_synapse_admin_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-synapse-admin.service wants matrix_synapse_admin_systemd_wanted_services_list: [] -# Controls whether the matrix-synapse-admin container exposes its HTTP port (tcp/80 in the container). +# Specifies the value of the `X-XSS-Protection` header +# Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. # -# Takes an ":" or "" value (e.g. "127.0.0.1:8766"), or empty string to not expose. -matrix_synapse_admin_container_http_host_bind_port: '' +# Learn more about it is here: +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection +# - https://portswigger.net/web-security/cross-site-scripting/reflected +matrix_synapse_admin_http_header_xss_protection: "1; mode=block" + +# Specifies the value of the `X-Frame-Options` header which controls whether framing can happen. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options +matrix_synapse_admin_http_header_frame_options: SAMEORIGIN + +# Specifies the value of the `X-Content-Type-Options` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options +matrix_synapse_admin_http_header_content_type_options: nosniff + +# Specifies the value of the `Content-Security-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy +matrix_synapse_admin_http_header_content_security_policy: frame-ancestors 'self' + +# Specifies the value of the `Permission-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy +matrix_synapse_admin_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_synapse_admin_floc_optout_enabled else '' }}" + +# Specifies the value of the `Strict-Transport-Security` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +matrix_synapse_admin_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_synapse_admin_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" + +# Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses +# +# Learn more about what it is here: +# - https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea +# - https://paramdeo.com/blog/opting-your-website-out-of-googles-floc-network +# - https://amifloced.org/ +# +# Of course, a better solution is to just stop using browsers (like Chrome), which participate in such tracking practices. +# See: `matrix_synapse_admin_content_permission_policy` +matrix_synapse_admin_floc_optout_enabled: true + +# Controls if HSTS preloading is enabled +# +# In its strongest and recommended form, the [HSTS policy](https://www.chromium.org/hsts) includes all subdomains, and +# indicates a willingness to be "preloaded" into browsers: +# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload` +# For more information visit: +# - https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +# - https://hstspreload.org/#opt-in +# See: `matrix_synapse_admin_http_header_strict_transport_security` +matrix_synapse_admin_hsts_preload_enabled: false # The path at which Synapse Admin will be exposed on `matrix.DOMAIN` # (only applies when matrix-nginx-proxy is used). diff --git a/roles/custom/matrix-synapse-admin/templates/labels.j2 b/roles/custom/matrix-synapse-admin/templates/labels.j2 index c70892a8..b754f6b8 100644 --- a/roles/custom/matrix-synapse-admin/templates/labels.j2 +++ b/roles/custom/matrix-synapse-admin/templates/labels.j2 @@ -18,6 +18,13 @@ traefik.http.middlewares.matrix-synapse-admin-strip-prefix.stripprefix.prefixes= {% set middlewares = middlewares + ['matrix-synapse-admin-strip-prefix'] %} {% endif %} +{% if matrix_synapse_admin_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_synapse_admin_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-synapse-admin-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-synapse-admin-add-headers'] %} +{% endif %} + traefik.http.routers.matrix-synapse-admin.rule={{ matrix_synapse_admin_container_labels_traefik_rule }} {% if middlewares | length > 0 %} traefik.http.routers.matrix-synapse-admin.middlewares={{ middlewares | join(',') }} From 5c7cd7068480e9e0f07c6243e80fe53a6cad37e9 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 08:51:20 +0200 Subject: [PATCH 146/382] Make use of the existing matrix_synapse_admin_public_endpoint variable --- roles/custom/matrix-synapse-admin/defaults/main.yml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index f150bd35..07a91833 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -43,7 +43,7 @@ matrix_synapse_admin_container_labels_traefik_enabled: true matrix_synapse_admin_container_labels_traefik_docker_network: "{{ matrix_synapse_admin_container_network }}" matrix_synapse_admin_container_labels_traefik_hostname: "{{ matrix_server_fqn_matrix }}" # The path prefix must either be `/` or not end with a slash (e.g. `/synapse-admin`). -matrix_synapse_admin_container_labels_traefik_path_prefix: /synapse-admin +matrix_synapse_admin_container_labels_traefik_path_prefix: "{{ matrix_synapse_admin_public_endpoint }}" matrix_synapse_admin_container_labels_traefik_rule: "Host(`{{ matrix_synapse_admin_container_labels_traefik_hostname }}`){% if matrix_synapse_admin_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }}`){% endif %}" matrix_synapse_admin_container_labels_traefik_entrypoints: web-secure matrix_synapse_admin_container_labels_traefik_tls: "{{ matrix_synapse_admin_container_labels_traefik_entrypoints != 'web' }}" @@ -130,6 +130,9 @@ matrix_synapse_admin_floc_optout_enabled: true # See: `matrix_synapse_admin_http_header_strict_transport_security` matrix_synapse_admin_hsts_preload_enabled: false -# The path at which Synapse Admin will be exposed on `matrix.DOMAIN` -# (only applies when matrix-nginx-proxy is used). +# The path at which Synapse Admin will be exposed on `matrix.DOMAIN` when matrix-nginx-proxy is used. +# A path of `/` is likely not a good choice when matrix-nginx-proxy is used. +# +# If Traefik is used, the hostname name is configurable - see `matrix_synapse_admin_container_labels_traefik_hostname`. +# This value must either be `/` or not end with a slash (e.g. `/synapse-admin`). matrix_synapse_admin_public_endpoint: /synapse-admin From 799cbb44fbaf800840ed5612a4a2ddc476ac98f4 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 09:04:50 +0200 Subject: [PATCH 147/382] Add the ability to control (Traefik) routing priority for Element and synapse-admin This may proof useful to someone in the future. --- roles/custom/matrix-client-element/defaults/main.yml | 1 + roles/custom/matrix-client-element/templates/labels.j2 | 3 +++ roles/custom/matrix-synapse-admin/defaults/main.yml | 1 + roles/custom/matrix-synapse-admin/templates/labels.j2 | 3 +++ 4 files changed, 8 insertions(+) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 500c5b47..041fb033 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -41,6 +41,7 @@ matrix_client_element_container_labels_traefik_hostname: "{{ matrix_server_fqn_e # The path prefix must either be `/` or not end with a slash (e.g. `/element`). matrix_client_element_container_labels_traefik_path_prefix: / matrix_client_element_container_labels_traefik_rule: "Host(`{{ matrix_client_element_container_labels_traefik_hostname }}`){% if matrix_client_element_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_client_element_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_client_element_container_labels_traefik_priority: 0 matrix_client_element_container_labels_traefik_entrypoints: web-secure matrix_client_element_container_labels_traefik_tls: "{{ matrix_client_element_container_labels_traefik_entrypoints != 'web' }}" matrix_client_element_container_labels_traefik_tls_certResolver: default # noqa var-naming diff --git a/roles/custom/matrix-client-element/templates/labels.j2 b/roles/custom/matrix-client-element/templates/labels.j2 index 33fb28e1..de67299e 100644 --- a/roles/custom/matrix-client-element/templates/labels.j2 +++ b/roles/custom/matrix-client-element/templates/labels.j2 @@ -26,6 +26,9 @@ traefik.http.middlewares.matrix-client-element-add-headers.headers.customrespons {% endif %} traefik.http.routers.matrix-client-element.rule={{ matrix_client_element_container_labels_traefik_rule }} +{% if matrix_client_element_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-client-element.priority={{ matrix_client_element_container_labels_traefik_priority }} +{% endif %} traefik.http.routers.matrix-client-element.service=matrix-client-element {% if middlewares | length > 0 %} traefik.http.routers.matrix-client-element.middlewares={{ middlewares | join(',') }} diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 07a91833..1d655d73 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -45,6 +45,7 @@ matrix_synapse_admin_container_labels_traefik_hostname: "{{ matrix_server_fqn_ma # The path prefix must either be `/` or not end with a slash (e.g. `/synapse-admin`). matrix_synapse_admin_container_labels_traefik_path_prefix: "{{ matrix_synapse_admin_public_endpoint }}" matrix_synapse_admin_container_labels_traefik_rule: "Host(`{{ matrix_synapse_admin_container_labels_traefik_hostname }}`){% if matrix_synapse_admin_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }}`){% endif %}" +matrix_synapse_admin_container_labels_traefik_priority: 0 matrix_synapse_admin_container_labels_traefik_entrypoints: web-secure matrix_synapse_admin_container_labels_traefik_tls: "{{ matrix_synapse_admin_container_labels_traefik_entrypoints != 'web' }}" matrix_synapse_admin_container_labels_traefik_tls_certResolver: default # noqa var-naming diff --git a/roles/custom/matrix-synapse-admin/templates/labels.j2 b/roles/custom/matrix-synapse-admin/templates/labels.j2 index b754f6b8..946fd6c9 100644 --- a/roles/custom/matrix-synapse-admin/templates/labels.j2 +++ b/roles/custom/matrix-synapse-admin/templates/labels.j2 @@ -26,6 +26,9 @@ traefik.http.middlewares.matrix-synapse-admin-add-headers.headers.customresponse {% endif %} traefik.http.routers.matrix-synapse-admin.rule={{ matrix_synapse_admin_container_labels_traefik_rule }} +{% if matrix_synapse_admin_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-synapse-admin.priority={{ matrix_synapse_admin_container_labels_traefik_priority }} +{% endif %} {% if middlewares | length > 0 %} traefik.http.routers.matrix-synapse-admin.middlewares={{ middlewares | join(',') }} {% endif %} From 64e2b26ed55fd6ee1c1ffc62a37cb09e452fe7ec Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 09:49:22 +0200 Subject: [PATCH 148/382] Fix Hydrogen failing to start We were mounting our own configuration to `/usr/share/nginx/html/config.json`, which is a symlink to `/tmp/config.json`. So we effectively mount our file to `/tmp/config.json`. When starting: - if Hydrogen sees a `CONFIG_OVERRIDE` environment variable, it will try to save it into our read-only config file and fail. - if Hydrogen doesn't see a `CONFIG_OVERRIDE` environment variable (the path we go through, because we don't pass such a variable), it will try to copy its bundled configuration (`/config.json.bundled`) to `/tmp/config.json`. Because our configuration is mounted as read-only, it will fail. In both cases, it will fail with: > cp: can't create '/tmp/config.json': File exists Source: https://github.com/vector-im/hydrogen-web/blob/3720de36bbee8609ca7cf625e7b72dc44609e393/docker/dynamic-config.sh We work around this by mounting our configuration on top of the bundled one (`/config.json.bundled`). We then let Hydrogen's startup script copy it to `/tmp/config.json` (a tmpfs we've mounted into the container) and use it from there. --- .../templates/systemd/matrix-client-hydrogen.service.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 b/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 index d8a3fb98..e09978ea 100644 --- a/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 +++ b/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 @@ -24,7 +24,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endif %} --tmpfs=/tmp:rw,noexec,nosuid,size=10m \ --mount type=bind,src={{ matrix_client_hydrogen_data_path }}/nginx.conf,dst=/etc/nginx/nginx.conf,ro \ - --mount type=bind,src={{ matrix_client_hydrogen_data_path }}/config.json,dst=/usr/share/nginx/html/config.json,ro \ + --mount type=bind,src={{ matrix_client_hydrogen_data_path }}/config.json,dst=/config.json.bundled,ro \ {% for arg in matrix_client_hydrogen_container_extra_arguments %} {{ arg }} \ {% endfor %} From 6a52be79877daf9d320bb5ef28389977d1ec8d31 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 09:58:35 +0200 Subject: [PATCH 149/382] Add (native) Traefik support to matrix-client-hydrogen Previously, it had to go through matrix-nginx-proxy. It's exposed to Traefik directly via container labels now Serving at a path other than `/` doesn't work well yet. --- group_vars/matrix_servers | 8 ++ .../matrix-client-hydrogen/defaults/main.yml | 99 +++++++++++++++++++ .../tasks/setup_install.yml | 7 +- .../tasks/validate_config.yml | 31 ++++++ .../templates/labels.j2 | 45 +++++++++ .../systemd/matrix-client-hydrogen.service.j2 | 14 ++- 6 files changed, 201 insertions(+), 3 deletions(-) create mode 100644 roles/custom/matrix-client-hydrogen/templates/labels.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index ca046040..1a131a28 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2987,6 +2987,14 @@ matrix_client_hydrogen_container_image_self_build: "{{ matrix_architecture not i # the HTTP port to the local host. matrix_client_hydrogen_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8768') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_client_hydrogen_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-client-hydrogen' }}" + +matrix_client_hydrogen_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + +matrix_client_hydrogen_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" +matrix_client_hydrogen_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_client_hydrogen_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" + matrix_client_hydrogen_default_hs_url: "{{ matrix_homeserver_url }}" matrix_client_hydrogen_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" diff --git a/roles/custom/matrix-client-hydrogen/defaults/main.yml b/roles/custom/matrix-client-hydrogen/defaults/main.yml index 6ee304cf..d429707c 100644 --- a/roles/custom/matrix-client-hydrogen/defaults/main.yml +++ b/roles/custom/matrix-client-hydrogen/defaults/main.yml @@ -14,17 +14,116 @@ matrix_client_hydrogen_docker_image_force_pull: "{{ matrix_client_hydrogen_docke matrix_client_hydrogen_data_path: "{{ matrix_base_data_path }}/client-hydrogen" matrix_client_hydrogen_docker_src_files_path: "{{ matrix_client_hydrogen_data_path }}/docker-src" +# The base container network +matrix_client_hydrogen_container_network: matrix-client-hydrogen + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to a reverse proxy, which runs in a different container network. +matrix_client_hydrogen_container_additional_networks: [] + # Controls whether the container exposes its HTTP port (tcp/8080 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:8768"), or empty string to not expose. matrix_client_hydrogen_container_http_host_bind_port: '' +# matrix_client_hydrogen_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_client_hydrogen_container_labels_additional_labels`. +matrix_client_hydrogen_container_labels_traefik_enabled: true +matrix_client_hydrogen_container_labels_traefik_docker_network: "{{ matrix_client_hydrogen_container_network }}" +matrix_client_hydrogen_container_labels_traefik_hostname: "{{ matrix_server_fqn_element }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/hydrogen`). +# For now, only `/` is supported due to problems with serving Hydrogen at a subpath. +matrix_client_hydrogen_container_labels_traefik_path_prefix: / +matrix_client_hydrogen_container_labels_traefik_rule: "Host(`{{ matrix_client_hydrogen_container_labels_traefik_hostname }}`){% if matrix_client_hydrogen_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_client_hydrogen_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_client_hydrogen_container_labels_traefik_priority: 0 +matrix_client_hydrogen_container_labels_traefik_entrypoints: web-secure +matrix_client_hydrogen_container_labels_traefik_tls: "{{ matrix_client_hydrogen_container_labels_traefik_entrypoints != 'web' }}" +matrix_client_hydrogen_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_client_hydrogen_container_labels_traefik_additional_response_headers_custom` +matrix_client_hydrogen_container_labels_traefik_additional_response_headers: "{{ matrix_client_hydrogen_container_labels_traefik_additional_response_headers_auto | combine(matrix_client_hydrogen_container_labels_traefik_additional_response_headers_custom) }}" +matrix_client_hydrogen_container_labels_traefik_additional_response_headers_auto: | + {{ + {} + | combine ({'X-XSS-Protection': matrix_client_hydrogen_http_header_xss_protection} if matrix_client_hydrogen_http_header_xss_protection else {}) + | combine ({'X-Frame-Options': matrix_client_hydrogen_http_header_frame_options} if matrix_client_hydrogen_http_header_frame_options else {}) + | combine ({'X-Content-Type-Options': matrix_client_hydrogen_http_header_content_type_options} if matrix_client_hydrogen_http_header_content_type_options else {}) + | combine ({'Content-Security-Policy': matrix_client_hydrogen_http_header_content_security_policy} if matrix_client_hydrogen_http_header_content_security_policy else {}) + | combine ({'Permission-Policy': matrix_client_hydrogen_http_header_content_permission_policy} if matrix_client_hydrogen_http_header_content_permission_policy else {}) + | combine ({'Strict-Transport-Security': matrix_client_hydrogen_http_header_strict_transport_security} if matrix_client_hydrogen_http_header_strict_transport_security and matrix_client_hydrogen_container_labels_traefik_tls else {}) + }} +matrix_client_hydrogen_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_client_hydrogen_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_client_hydrogen_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_client_hydrogen_container_labels_additional_labels: '' + # A list of extra arguments to pass to the container matrix_client_hydrogen_container_extra_arguments: [] # List of systemd services that matrix-client-hydrogen.service depends on matrix_client_hydrogen_systemd_required_services_list: ['docker.service'] +# Specifies the value of the `X-XSS-Protection` header +# Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. +# +# Learn more about it is here: +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection +# - https://portswigger.net/web-security/cross-site-scripting/reflected +matrix_client_hydrogen_http_header_xss_protection: "1; mode=block" + +# Specifies the value of the `X-Frame-Options` header which controls whether framing can happen. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options +matrix_client_hydrogen_http_header_frame_options: SAMEORIGIN + +# Specifies the value of the `X-Content-Type-Options` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options +matrix_client_hydrogen_http_header_content_type_options: nosniff + +# Specifies the value of the `Content-Security-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy +matrix_client_hydrogen_http_header_content_security_policy: frame-ancestors 'self' + +# Specifies the value of the `Permission-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy +matrix_client_hydrogen_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_client_hydrogen_floc_optout_enabled else '' }}" + +# Specifies the value of the `Strict-Transport-Security` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +matrix_client_hydrogen_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_client_hydrogen_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" + +# Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses +# +# Learn more about what it is here: +# - https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea +# - https://paramdeo.com/blog/opting-your-website-out-of-googles-floc-network +# - https://amifloced.org/ +# +# Of course, a better solution is to just stop using browsers (like Chrome), which participate in such tracking practices. +# See: `matrix_client_hydrogen_content_permission_policy` +matrix_client_hydrogen_floc_optout_enabled: true + +# Controls if HSTS preloading is enabled +# +# In its strongest and recommended form, the [HSTS policy](https://www.chromium.org/hsts) includes all subdomains, and +# indicates a willingness to be "preloaded" into browsers: +# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload` +# For more information visit: +# - https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +# - https://hstspreload.org/#opt-in +# See: `matrix_client_hydrogen_http_header_strict_transport_security` +matrix_client_hydrogen_hsts_preload_enabled: false + # Controls whether the self-check feature should validate SSL certificates. matrix_client_hydrogen_self_check_validate_certificates: true diff --git a/roles/custom/matrix-client-hydrogen/tasks/setup_install.yml b/roles/custom/matrix-client-hydrogen/tasks/setup_install.yml index 5fe4cead..0e114804 100644 --- a/roles/custom/matrix-client-hydrogen/tasks/setup_install.yml +++ b/roles/custom/matrix-client-hydrogen/tasks/setup_install.yml @@ -52,7 +52,7 @@ group: "{{ matrix_user_groupname }}" with_items: - {src: "{{ role_path }}/templates/nginx.conf.j2", name: "nginx.conf"} - when: "item.src is not none" + - {src: "{{ role_path }}/templates/labels.j2", name: "labels"} - name: Ensure Hydrogen Docker image is built community.docker.docker_image: @@ -65,6 +65,11 @@ pull: true when: "matrix_client_hydrogen_container_image_self_build | bool" +- name: Ensure Hydrogen container network is created + community.general.docker_network: + name: "{{ matrix_client_hydrogen_container_network }}" + driver: bridge + - name: Ensure matrix-client-hydrogen.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-client-hydrogen.service.j2" diff --git a/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml b/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml index 65b0e14f..bc772594 100644 --- a/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml +++ b/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml @@ -7,3 +7,34 @@ when: "(vars[item] == '' or vars[item] is none) and matrix_client_hydrogen_container_image_self_build | bool" with_items: - "matrix_client_hydrogen_default_hs_url" + +- when: matrix_client_hydrogen_container_labels_traefik_enabled | bool + block: + - name: Fail if required matrix-client-hydrogen Traefik settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_client_hydrogen_container_labels_traefik_hostname + - matrix_client_hydrogen_container_labels_traefik_path_prefix + + # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). + # Knowing that `matrix_client_hydrogen_container_labels_traefik_path_prefix` does not end with a slash + # ensures we know how to set these routes up without having to do "does it end with a slash" checks elsewhere. + - name: Fail if matrix_client_hydrogen_container_labels_traefik_path_prefix ends with a slash + ansible.builtin.fail: + msg: >- + matrix_client_hydrogen_container_labels_traefik_path_prefix (`{{ matrix_client_hydrogen_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/hydrogen`). + when: "matrix_client_hydrogen_container_labels_traefik_path_prefix != '/' and matrix_client_hydrogen_container_labels_traefik_path_prefix[-1] == '/'" + + # For example, we're getting errors like this: + # > main.js:51 Failed to load stylesheet from https://DOMAIN/assets/theme-element-light.5316e268.css: + # despite having set a prefix of `/hydrogen`. + # This is yet to be investigated and workaround around. + - name: Fail if matrix_client_hydrogen_container_labels_traefik_path_prefix is not / + ansible.builtin.fail: + msg: >- + matrix_client_hydrogen_container_labels_traefik_path_prefix can only be set to `/` right now. + Serving Hydrogen from another path doesn't work just yet. + when: "matrix_client_hydrogen_container_labels_traefik_path_prefix != '/'" diff --git a/roles/custom/matrix-client-hydrogen/templates/labels.j2 b/roles/custom/matrix-client-hydrogen/templates/labels.j2 new file mode 100644 index 00000000..71c3ca59 --- /dev/null +++ b/roles/custom/matrix-client-hydrogen/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_client_hydrogen_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_client_hydrogen_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_client_hydrogen_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_client_hydrogen_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-client-hydrogen-slashless-redirect.redirectregex.regex=({{ matrix_client_hydrogen_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-client-hydrogen-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-client-hydrogen-slashless-redirect'] %} +{% endif %} + +{% if matrix_client_hydrogen_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-client-hydrogen-strip-prefix.stripprefix.prefixes={{ matrix_client_hydrogen_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-client-hydrogen-strip-prefix'] %} +{% endif %} + +{% if matrix_client_hydrogen_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_client_hydrogen_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-client-hydrogen-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-client-hydrogen-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-client-hydrogen.rule={{ matrix_client_hydrogen_container_labels_traefik_rule }} +{% if matrix_client_hydrogen_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-client-hydrogen.priority={{ matrix_client_hydrogen_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-client-hydrogen.service=matrix-client-hydrogen +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-client-hydrogen.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-client-hydrogen.entrypoints={{ matrix_client_hydrogen_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-client-hydrogen.tls={{ matrix_client_hydrogen_container_labels_traefik_tls | to_json }} +{% if matrix_client_hydrogen_container_labels_traefik_tls %} +traefik.http.routers.matrix-client-hydrogen.tls.certResolver={{ matrix_client_hydrogen_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-client-hydrogen.loadbalancer.server.port=8080 +{% endif %} + +{{ matrix_client_hydrogen_container_labels_additional_labels }} diff --git a/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 b/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 index e09978ea..1f1f0dc9 100644 --- a/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 +++ b/roles/custom/matrix-client-hydrogen/templates/systemd/matrix-client-hydrogen.service.j2 @@ -13,15 +13,18 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-client-hydrogen 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-client-hydrogen 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-client-hydrogen \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-client-hydrogen \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_client_hydrogen_container_network }} \ {% if matrix_client_hydrogen_container_http_host_bind_port %} -p {{ matrix_client_hydrogen_container_http_host_bind_port }}:8080 \ {% endif %} + --label-file={{ matrix_client_hydrogen_data_path }}/labels \ --tmpfs=/tmp:rw,noexec,nosuid,size=10m \ --mount type=bind,src={{ matrix_client_hydrogen_data_path }}/nginx.conf,dst=/etc/nginx/nginx.conf,ro \ --mount type=bind,src={{ matrix_client_hydrogen_data_path }}/config.json,dst=/config.json.bundled,ro \ @@ -30,8 +33,15 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endfor %} {{ matrix_client_hydrogen_docker_image }} +{% for network in matrix_client_hydrogen_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-client-hydrogen +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-client-hydrogen + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-client-hydrogen 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-client-hydrogen 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-client-hydrogen From eb7292f274bb7c4cfa9ce26ff72aa92da0018154 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 10:56:16 +0200 Subject: [PATCH 150/382] Add matrix_client_hydrogen_hostname and fix Hydrogen serving at non-root-path --- .../matrix-client-hydrogen/defaults/main.yml | 18 ++++++++++++++---- .../tasks/validate_config.yml | 11 ----------- .../templates/config.json.j2 | 2 +- .../matrix-synapse-admin/defaults/main.yml | 2 +- 4 files changed, 16 insertions(+), 17 deletions(-) diff --git a/roles/custom/matrix-client-hydrogen/defaults/main.yml b/roles/custom/matrix-client-hydrogen/defaults/main.yml index d429707c..6ead5f44 100644 --- a/roles/custom/matrix-client-hydrogen/defaults/main.yml +++ b/roles/custom/matrix-client-hydrogen/defaults/main.yml @@ -33,10 +33,9 @@ matrix_client_hydrogen_container_http_host_bind_port: '' # To inject your own other container labels, see `matrix_client_hydrogen_container_labels_additional_labels`. matrix_client_hydrogen_container_labels_traefik_enabled: true matrix_client_hydrogen_container_labels_traefik_docker_network: "{{ matrix_client_hydrogen_container_network }}" -matrix_client_hydrogen_container_labels_traefik_hostname: "{{ matrix_server_fqn_element }}" +matrix_client_hydrogen_container_labels_traefik_hostname: "{{ matrix_client_hydrogen_hostname }}" # The path prefix must either be `/` or not end with a slash (e.g. `/hydrogen`). -# For now, only `/` is supported due to problems with serving Hydrogen at a subpath. -matrix_client_hydrogen_container_labels_traefik_path_prefix: / +matrix_client_hydrogen_container_labels_traefik_path_prefix: "{{ matrix_client_hydrogen_path_prefix }}" matrix_client_hydrogen_container_labels_traefik_rule: "Host(`{{ matrix_client_hydrogen_container_labels_traefik_hostname }}`){% if matrix_client_hydrogen_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_client_hydrogen_container_labels_traefik_path_prefix }}`){% endif %}" matrix_client_hydrogen_container_labels_traefik_priority: 0 matrix_client_hydrogen_container_labels_traefik_entrypoints: web-secure @@ -124,6 +123,18 @@ matrix_client_hydrogen_floc_optout_enabled: true # See: `matrix_client_hydrogen_http_header_strict_transport_security` matrix_client_hydrogen_hsts_preload_enabled: false +# The hostname at which Hydrogen is served. +# Only works with with Traefik reverse-proxying. +# For matrix-nginx-proxy, `matrix_server_fqn_hydrogen` is used and this variable has no effect. +matrix_client_hydrogen_hostname: "{{ matrix_server_fqn_hydrogen }}" + +# The path at which Hydrogen is exposed. +# When matrix-nginx-proxy is used, setting this to values other than `/` will cause configuration mismatches and trouble. +# +# If Traefik is used, the hostname is also configurable - see `matrix_client_hydrogen_container_labels_traefik_hostname`. +# This value must either be `/` or not end with a slash (e.g. `/hydrogen`). +matrix_client_hydrogen_path_prefix: / + # Controls whether the self-check feature should validate SSL certificates. matrix_client_hydrogen_self_check_validate_certificates: true @@ -135,7 +146,6 @@ matrix_client_hydrogen_push: matrix_client_hydrogen_default_hs_url: "" matrix_client_hydrogen_bugReportEndpointUrl: "https://element.io/bugreports/submit" # noqa var-naming - # Default Hydrogen configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml b/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml index bc772594..dd880ddb 100644 --- a/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml +++ b/roles/custom/matrix-client-hydrogen/tasks/validate_config.yml @@ -27,14 +27,3 @@ msg: >- matrix_client_hydrogen_container_labels_traefik_path_prefix (`{{ matrix_client_hydrogen_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/hydrogen`). when: "matrix_client_hydrogen_container_labels_traefik_path_prefix != '/' and matrix_client_hydrogen_container_labels_traefik_path_prefix[-1] == '/'" - - # For example, we're getting errors like this: - # > main.js:51 Failed to load stylesheet from https://DOMAIN/assets/theme-element-light.5316e268.css: - # despite having set a prefix of `/hydrogen`. - # This is yet to be investigated and workaround around. - - name: Fail if matrix_client_hydrogen_container_labels_traefik_path_prefix is not / - ansible.builtin.fail: - msg: >- - matrix_client_hydrogen_container_labels_traefik_path_prefix can only be set to `/` right now. - Serving Hydrogen from another path doesn't work just yet. - when: "matrix_client_hydrogen_container_labels_traefik_path_prefix != '/'" diff --git a/roles/custom/matrix-client-hydrogen/templates/config.json.j2 b/roles/custom/matrix-client-hydrogen/templates/config.json.j2 index b6b1b9be..e503c105 100644 --- a/roles/custom/matrix-client-hydrogen/templates/config.json.j2 +++ b/roles/custom/matrix-client-hydrogen/templates/config.json.j2 @@ -3,7 +3,7 @@ "defaultHomeServer": {{ matrix_client_hydrogen_default_hs_url | string | to_json }}, "bugReportEndpointUrl": {{ matrix_client_hydrogen_bugReportEndpointUrl | to_json }}, "themeManifests": [ - "assets/theme-element.json" + "{{ matrix_client_hydrogen_path_prefix }}assets/theme-element.json" ], "defaultTheme": { "light": "element-light", diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 1d655d73..4345a026 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -134,6 +134,6 @@ matrix_synapse_admin_hsts_preload_enabled: false # The path at which Synapse Admin will be exposed on `matrix.DOMAIN` when matrix-nginx-proxy is used. # A path of `/` is likely not a good choice when matrix-nginx-proxy is used. # -# If Traefik is used, the hostname name is configurable - see `matrix_synapse_admin_container_labels_traefik_hostname`. +# If Traefik is used, the hostname is also configurable - see `matrix_synapse_admin_container_labels_traefik_hostname`. # This value must either be `/` or not end with a slash (e.g. `/synapse-admin`). matrix_synapse_admin_public_endpoint: /synapse-admin From 2e74187050b87d2835e385bf3af245d9767efd2b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 11:02:18 +0200 Subject: [PATCH 151/382] Add matrix_client_element_hostname and matrix_client_element_path_prefix variables --- .../matrix-client-element/defaults/main.yml | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 041fb033..bacaa786 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -37,9 +37,9 @@ matrix_client_element_container_http_host_bind_port: '' # To inject your own other container labels, see `matrix_client_element_container_labels_additional_labels`. matrix_client_element_container_labels_traefik_enabled: true matrix_client_element_container_labels_traefik_docker_network: "{{ matrix_client_element_container_network }}" -matrix_client_element_container_labels_traefik_hostname: "{{ matrix_server_fqn_element }}" +matrix_client_element_container_labels_traefik_hostname: "{{ matrix_client_element_hostname }}" # The path prefix must either be `/` or not end with a slash (e.g. `/element`). -matrix_client_element_container_labels_traefik_path_prefix: / +matrix_client_element_container_labels_traefik_path_prefix: "{{ matrix_client_element_path_prefix }}" matrix_client_element_container_labels_traefik_rule: "Host(`{{ matrix_client_element_container_labels_traefik_hostname }}`){% if matrix_client_element_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_client_element_container_labels_traefik_path_prefix }}`){% endif %}" matrix_client_element_container_labels_traefik_priority: 0 matrix_client_element_container_labels_traefik_entrypoints: web-secure @@ -127,6 +127,18 @@ matrix_client_element_floc_optout_enabled: true # See: `matrix_client_element_http_header_strict_transport_security` matrix_client_element_hsts_preload_enabled: false +# The hostname at which Element is served. +# Only works with with Traefik reverse-proxying. +# For matrix-nginx-proxy, `matrix_server_fqn_element` is used and this variable has no effect. +matrix_client_element_hostname: "{{ matrix_server_fqn_element }}" + +# The path at which Element is exposed. +# When matrix-nginx-proxy is used, setting this to values other than `/` will cause configuration mismatches and trouble. +# +# If Traefik is used, the hostname is also configurable - see `matrix_client_element_container_labels_traefik_hostname`. +# This value must either be `/` or not end with a slash (e.g. `/element`). +matrix_client_element_path_prefix: / + # Element config.json customizations matrix_client_element_default_server_name: "{{ matrix_domain }}" matrix_client_element_default_hs_url: "" From 3bace0c7b9d43ce860078a20a24a9c7a2c19114b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 11:05:39 +0200 Subject: [PATCH 152/382] Add matrix_synapse_admin_hostname and rename matrix_synapse_admin_public_endpoint (to matrix_synapse_admin_path_prefix) --- docs/configuring-playbook-synapse-admin.md | 28 ------------------- .../matrix-synapse-admin/defaults/main.yml | 15 ++++++---- .../tasks/inject_into_nginx_proxy.yml | 6 ++-- .../tasks/validate_config.yml | 1 + 4 files changed, 14 insertions(+), 36 deletions(-) diff --git a/docs/configuring-playbook-synapse-admin.md b/docs/configuring-playbook-synapse-admin.md index ad1bda02..1099553b 100644 --- a/docs/configuring-playbook-synapse-admin.md +++ b/docs/configuring-playbook-synapse-admin.md @@ -35,34 +35,6 @@ To use Synapse Admin, you need to have [registered at least one administrator ac The Homeserver URL to use on Synapse Admin's login page is: `https://matrix.DOMAIN` -### Sample configuration for running behind Traefik 2.0 - -Below is a sample configuration for using this playbook with a [Traefik](https://traefik.io/) 2.0 reverse proxy. - -This an extension to Traefik config sample in [own-webserver-documentation](./configuring-playbook-own-webserver.md). - -```yaml -# Don't bind any HTTP or federation port to the host -# (Traefik will proxy directly into the containers) -matrix_synapse_admin_container_http_host_bind_port: "" - -matrix_synapse_admin_container_extra_arguments: - # May be unnecessary depending on Traefik config, but can't hurt - - '--label "traefik.enable=true"' - - # The Synapse Admin container will only receive traffic from this subdomain and path - - '--label "traefik.http.routers.matrix-synapse-admin.rule=(Host(`{{ matrix_server_fqn_matrix }}`) && Path(`{{matrix_synapse_admin_public_endpoint}}`))"' - - # (Define your entrypoint) - - '--label "traefik.http.routers.matrix-synapse-admin.entrypoints=web-secure"' - - # (The 'default' certificate resolver must be defined in Traefik config) - - '--label "traefik.http.routers.matrix-synapse-admin.tls.certResolver=default"' - - # The Synapse Admin container uses port 80 by default - - '--label "traefik.http.services.matrix-synapse-admin.loadbalancer.server.port=80"' -``` - ### Sample configuration for running behind Caddy v2 Below is a sample configuration for using this playbook with a [Caddy](https://caddyserver.com/v2) 2.0 reverse proxy (non-default configuration where `matrix-nginx-proxy` is disabled - `matrix_nginx_proxy_enabled: false`). diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 4345a026..91383a9e 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -41,9 +41,9 @@ matrix_synapse_admin_container_extra_arguments: [] # To inject your own other container labels, see `matrix_synapse_admin_container_labels_additional_labels`. matrix_synapse_admin_container_labels_traefik_enabled: true matrix_synapse_admin_container_labels_traefik_docker_network: "{{ matrix_synapse_admin_container_network }}" -matrix_synapse_admin_container_labels_traefik_hostname: "{{ matrix_server_fqn_matrix }}" +matrix_synapse_admin_container_labels_traefik_hostname: "{{ matrix_synapse_admin_hostname }}" # The path prefix must either be `/` or not end with a slash (e.g. `/synapse-admin`). -matrix_synapse_admin_container_labels_traefik_path_prefix: "{{ matrix_synapse_admin_public_endpoint }}" +matrix_synapse_admin_container_labels_traefik_path_prefix: "{{ matrix_synapse_admin_path_prefix }}" matrix_synapse_admin_container_labels_traefik_rule: "Host(`{{ matrix_synapse_admin_container_labels_traefik_hostname }}`){% if matrix_synapse_admin_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_synapse_admin_container_labels_traefik_path_prefix | quote }}`){% endif %}" matrix_synapse_admin_container_labels_traefik_priority: 0 matrix_synapse_admin_container_labels_traefik_entrypoints: web-secure @@ -131,9 +131,14 @@ matrix_synapse_admin_floc_optout_enabled: true # See: `matrix_synapse_admin_http_header_strict_transport_security` matrix_synapse_admin_hsts_preload_enabled: false -# The path at which Synapse Admin will be exposed on `matrix.DOMAIN` when matrix-nginx-proxy is used. -# A path of `/` is likely not a good choice when matrix-nginx-proxy is used. +# The hostname at which Synapse Admin is served. +# Only works with with Traefik reverse-proxying. +# For matrix-nginx-proxy, `matrix_server_fqn_matrix` is used and this variable has no effect. +matrix_synapse_admin_hostname: "{{ matrix_server_fqn_matrix }}" + +# The path at which Synapse Admin is exposed. +# When matrix-nginx-proxy is used, setting this to values other than `/` will cause configuration mismatches and trouble. # # If Traefik is used, the hostname is also configurable - see `matrix_synapse_admin_container_labels_traefik_hostname`. # This value must either be `/` or not end with a slash (e.g. `/synapse-admin`). -matrix_synapse_admin_public_endpoint: /synapse-admin +matrix_synapse_admin_path_prefix: /synapse-admin diff --git a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml index 6a4af859..a06f47a1 100644 --- a/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-synapse-admin/tasks/inject_into_nginx_proxy.yml @@ -12,9 +12,9 @@ - name: Generate Synapse Admin proxying configuration for matrix-nginx-proxy ansible.builtin.set_fact: matrix_synapse_admin_matrix_nginx_proxy_configuration: | - rewrite ^{{ matrix_synapse_admin_public_endpoint }}$ {{ matrix_nginx_proxy_x_forwarded_proto_value }}://$server_name{{ matrix_synapse_admin_public_endpoint }}/ permanent; + rewrite ^{{ matrix_synapse_admin_path_prefix }}$ {{ matrix_nginx_proxy_x_forwarded_proto_value }}://$server_name{{ matrix_synapse_admin_path_prefix }}/ permanent; - location ~ ^{{ matrix_synapse_admin_public_endpoint }}/(.*) { + location ~ ^{{ matrix_synapse_admin_path_prefix }}/(.*) { {% if matrix_nginx_proxy_enabled | default(False) %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; @@ -40,7 +40,7 @@ msg: >- NOTE: You've enabled the Synapse Admin tool but are not using the matrix-nginx-proxy reverse proxy. - Please make sure that you're proxying the `{{ matrix_synapse_admin_public_endpoint }}` + Please make sure that you're proxying the `{{ matrix_synapse_admin_path_prefix }}` URL endpoint to the matrix-synapse-admin container. You can expose the container's port using the `matrix_synapse_admin_container_http_host_bind_port` variable. when: "not matrix_nginx_proxy_enabled | default(False) | bool" diff --git a/roles/custom/matrix-synapse-admin/tasks/validate_config.yml b/roles/custom/matrix-synapse-admin/tasks/validate_config.yml index 48243555..d0281986 100644 --- a/roles/custom/matrix-synapse-admin/tasks/validate_config.yml +++ b/roles/custom/matrix-synapse-admin/tasks/validate_config.yml @@ -10,6 +10,7 @@ - {'old': 'matrix_synapse_admin_docker_repo', 'new': 'matrix_synapse_admin_container_self_build_repo'} - {'old': 'matrix_synapse_admin_container_self_build', 'new': 'matrix_synapse_admin_container_image_self_build'} - {'old': 'matrix_synapse_admin_container_self_build_repo', 'new': 'matrix_synapse_admin_container_image_self_build_repo'} + - {'old': 'matrix_synapse_admin_public_endpoint', 'new': 'matrix_synapse_admin_path_prefix'} - when: matrix_synapse_admin_container_labels_traefik_enabled | bool block: From f28e7ef9c71f02c3935be0c0c8b3f21bcb69a26d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 11:29:53 +0200 Subject: [PATCH 153/382] Add (native) Traefik support to matrix-client-cinny Previously, it had to go through matrix-nginx-proxy. It's exposed to Traefik directly via container labels now --- group_vars/matrix_servers | 8 ++ .../matrix-client-cinny/defaults/main.yml | 111 ++++++++++++++++++ .../tasks/setup_install.yml | 7 +- .../tasks/validate_config.yml | 20 ++++ .../matrix-client-cinny/templates/labels.j2 | 45 +++++++ .../systemd/matrix-client-cinny.service.j2 | 14 ++- 6 files changed, 202 insertions(+), 3 deletions(-) create mode 100644 roles/custom/matrix-client-cinny/templates/labels.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 1a131a28..6e311db1 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3020,6 +3020,14 @@ matrix_client_cinny_container_image_self_build: "{{ matrix_architecture not in [ # the HTTP port to the local host. matrix_client_cinny_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8080') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_client_cinny_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-client-cinny' }}" + +matrix_client_cinny_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + +matrix_client_cinny_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" +matrix_client_cinny_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_client_cinny_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" + matrix_client_cinny_default_hs_url: "{{ matrix_homeserver_url }}" matrix_client_cinny_self_check_validate_certificates: "{{ false if matrix_playbook_ssl_retrieval_method == 'self-signed' else true }}" diff --git a/roles/custom/matrix-client-cinny/defaults/main.yml b/roles/custom/matrix-client-cinny/defaults/main.yml index 5b15b42c..5e5f216f 100644 --- a/roles/custom/matrix-client-cinny/defaults/main.yml +++ b/roles/custom/matrix-client-cinny/defaults/main.yml @@ -14,17 +14,128 @@ matrix_client_cinny_docker_image_force_pull: "{{ matrix_client_cinny_docker_imag matrix_client_cinny_data_path: "{{ matrix_base_data_path }}/client-cinny" matrix_client_cinny_docker_src_files_path: "{{ matrix_client_cinny_data_path }}/docker-src" +# The base container network +matrix_client_cinny_container_network: matrix-client-cinny + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to a reverse proxy, which runs in a different container network. +matrix_client_cinny_container_additional_networks: [] + # Controls whether the container exposes its HTTP port (tcp/8080 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:8768"), or empty string to not expose. matrix_client_cinny_container_http_host_bind_port: '' + +# matrix_client_cinny_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_client_cinny_container_labels_additional_labels`. +matrix_client_cinny_container_labels_traefik_enabled: true +matrix_client_cinny_container_labels_traefik_docker_network: "{{ matrix_client_cinny_container_network }}" +matrix_client_cinny_container_labels_traefik_hostname: "{{ matrix_client_cinny_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/cinny`). +matrix_client_cinny_container_labels_traefik_path_prefix: "{{ matrix_client_cinny_path_prefix }}" +matrix_client_cinny_container_labels_traefik_rule: "Host(`{{ matrix_client_cinny_container_labels_traefik_hostname }}`){% if matrix_client_cinny_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_client_cinny_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_client_cinny_container_labels_traefik_priority: 0 +matrix_client_cinny_container_labels_traefik_entrypoints: web-secure +matrix_client_cinny_container_labels_traefik_tls: "{{ matrix_client_cinny_container_labels_traefik_entrypoints != 'web' }}" +matrix_client_cinny_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_client_cinny_container_labels_traefik_additional_response_headers_custom` +matrix_client_cinny_container_labels_traefik_additional_response_headers: "{{ matrix_client_cinny_container_labels_traefik_additional_response_headers_auto | combine(matrix_client_cinny_container_labels_traefik_additional_response_headers_custom) }}" +matrix_client_cinny_container_labels_traefik_additional_response_headers_auto: | + {{ + {} + | combine ({'X-XSS-Protection': matrix_client_cinny_http_header_xss_protection} if matrix_client_cinny_http_header_xss_protection else {}) + | combine ({'X-Frame-Options': matrix_client_cinny_http_header_frame_options} if matrix_client_cinny_http_header_frame_options else {}) + | combine ({'X-Content-Type-Options': matrix_client_cinny_http_header_content_type_options} if matrix_client_cinny_http_header_content_type_options else {}) + | combine ({'Content-Security-Policy': matrix_client_cinny_http_header_content_security_policy} if matrix_client_cinny_http_header_content_security_policy else {}) + | combine ({'Permission-Policy': matrix_client_cinny_http_header_content_permission_policy} if matrix_client_cinny_http_header_content_permission_policy else {}) + | combine ({'Strict-Transport-Security': matrix_client_cinny_http_header_strict_transport_security} if matrix_client_cinny_http_header_strict_transport_security and matrix_client_cinny_container_labels_traefik_tls else {}) + }} +matrix_client_cinny_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_client_cinny_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_client_cinny_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_client_cinny_container_labels_additional_labels: '' + # A list of extra arguments to pass to the container matrix_client_cinny_container_extra_arguments: [] # List of systemd services that matrix-client-cinny.service depends on matrix_client_cinny_systemd_required_services_list: ['docker.service'] +# Specifies the value of the `X-XSS-Protection` header +# Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. +# +# Learn more about it is here: +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection +# - https://portswigger.net/web-security/cross-site-scripting/reflected +matrix_client_cinny_http_header_xss_protection: "1; mode=block" + +# Specifies the value of the `X-Frame-Options` header which controls whether framing can happen. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options +matrix_client_cinny_http_header_frame_options: SAMEORIGIN + +# Specifies the value of the `X-Content-Type-Options` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options +matrix_client_cinny_http_header_content_type_options: nosniff + +# Specifies the value of the `Content-Security-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy +matrix_client_cinny_http_header_content_security_policy: frame-ancestors 'self' + +# Specifies the value of the `Permission-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy +matrix_client_cinny_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_client_cinny_floc_optout_enabled else '' }}" + +# Specifies the value of the `Strict-Transport-Security` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +matrix_client_cinny_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_client_cinny_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" + +# Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses +# +# Learn more about what it is here: +# - https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea +# - https://paramdeo.com/blog/opting-your-website-out-of-googles-floc-network +# - https://amifloced.org/ +# +# Of course, a better solution is to just stop using browsers (like Chrome), which participate in such tracking practices. +# See: `matrix_client_cinny_content_permission_policy` +matrix_client_cinny_floc_optout_enabled: true + +# Controls if HSTS preloading is enabled +# +# In its strongest and recommended form, the [HSTS policy](https://www.chromium.org/hsts) includes all subdomains, and +# indicates a willingness to be "preloaded" into browsers: +# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload` +# For more information visit: +# - https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +# - https://hstspreload.org/#opt-in +# See: `matrix_client_cinny_http_header_strict_transport_security` +matrix_client_cinny_hsts_preload_enabled: false + +# The hostname at which Cinny is served. +# Only works with with Traefik reverse-proxying. +# For matrix-nginx-proxy, `matrix_server_fqn_cinny` is used and this variable has no effect. +matrix_client_cinny_hostname: "{{ matrix_server_fqn_cinny }}" + +# The path at which Cinny is exposed. +# When matrix-nginx-proxy is used, setting this to values other than `/` will cause configuration mismatches and trouble. +# +# If Traefik is used, the hostname is also configurable - see `matrix_client_cinny_container_labels_traefik_hostname`. +# This value must either be `/` or not end with a slash (e.g. `/cinny`). +matrix_client_cinny_path_prefix: / + # Controls whether the self-check feature should validate SSL certificates. matrix_client_cinny_self_check_validate_certificates: true diff --git a/roles/custom/matrix-client-cinny/tasks/setup_install.yml b/roles/custom/matrix-client-cinny/tasks/setup_install.yml index 46458f0b..e4eb7938 100644 --- a/roles/custom/matrix-client-cinny/tasks/setup_install.yml +++ b/roles/custom/matrix-client-cinny/tasks/setup_install.yml @@ -51,7 +51,7 @@ group: "{{ matrix_user_groupname }}" with_items: - {src: "{{ role_path }}/templates/nginx.conf.j2", name: "nginx.conf"} - when: "item.src is not none" + - {src: "{{ role_path }}/templates/labels.j2", name: "labels"} - name: Ensure Cinny Docker image is built community.docker.docker_image: @@ -64,6 +64,11 @@ pull: true when: "matrix_client_cinny_container_image_self_build | bool" +- name: Ensure Cinny container network is created + community.general.docker_network: + name: "{{ matrix_client_cinny_container_network }}" + driver: bridge + - name: Ensure matrix-client-cinny.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-client-cinny.service.j2" diff --git a/roles/custom/matrix-client-cinny/tasks/validate_config.yml b/roles/custom/matrix-client-cinny/tasks/validate_config.yml index 10f64fee..33387209 100644 --- a/roles/custom/matrix-client-cinny/tasks/validate_config.yml +++ b/roles/custom/matrix-client-cinny/tasks/validate_config.yml @@ -6,3 +6,23 @@ when: "vars[item] == '' or vars[item] is none" with_items: - "matrix_client_cinny_default_hs_url" + +- when: matrix_client_cinny_container_labels_traefik_enabled | bool + block: + - name: Fail if required matrix-client-cinny Traefik settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_client_cinny_container_labels_traefik_hostname + - matrix_client_cinny_container_labels_traefik_path_prefix + + # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). + # Knowing that `matrix_client_cinny_container_labels_traefik_path_prefix` does not end with a slash + # ensures we know how to set these routes up without having to do "does it end with a slash" checks elsewhere. + - name: Fail if matrix_client_cinny_container_labels_traefik_path_prefix ends with a slash + ansible.builtin.fail: + msg: >- + matrix_client_cinny_container_labels_traefik_path_prefix (`{{ matrix_client_cinny_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/cinny`). + when: "matrix_client_cinny_container_labels_traefik_path_prefix != '/' and matrix_client_cinny_container_labels_traefik_path_prefix[-1] == '/'" diff --git a/roles/custom/matrix-client-cinny/templates/labels.j2 b/roles/custom/matrix-client-cinny/templates/labels.j2 new file mode 100644 index 00000000..f1c4f49a --- /dev/null +++ b/roles/custom/matrix-client-cinny/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_client_cinny_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_client_cinny_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_client_cinny_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_client_cinny_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-client-cinny-slashless-redirect.redirectregex.regex=({{ matrix_client_cinny_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-client-cinny-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-client-cinny-slashless-redirect'] %} +{% endif %} + +{% if matrix_client_cinny_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-client-cinny-strip-prefix.stripprefix.prefixes={{ matrix_client_cinny_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-client-cinny-strip-prefix'] %} +{% endif %} + +{% if matrix_client_cinny_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_client_cinny_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-client-cinny-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-client-cinny-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-client-cinny.rule={{ matrix_client_cinny_container_labels_traefik_rule }} +{% if matrix_client_cinny_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-client-cinny.priority={{ matrix_client_cinny_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-client-cinny.service=matrix-client-cinny +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-client-cinny.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-client-cinny.entrypoints={{ matrix_client_cinny_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-client-cinny.tls={{ matrix_client_cinny_container_labels_traefik_tls | to_json }} +{% if matrix_client_cinny_container_labels_traefik_tls %} +traefik.http.routers.matrix-client-cinny.tls.certResolver={{ matrix_client_cinny_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-client-cinny.loadbalancer.server.port=8080 +{% endif %} + +{{ matrix_client_cinny_container_labels_additional_labels }} diff --git a/roles/custom/matrix-client-cinny/templates/systemd/matrix-client-cinny.service.j2 b/roles/custom/matrix-client-cinny/templates/systemd/matrix-client-cinny.service.j2 index b9a66c74..2c49a0f3 100644 --- a/roles/custom/matrix-client-cinny/templates/systemd/matrix-client-cinny.service.j2 +++ b/roles/custom/matrix-client-cinny/templates/systemd/matrix-client-cinny.service.j2 @@ -13,15 +13,18 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-client-cinny 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-client-cinny 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-client-cinny \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-client-cinny \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_client_cinny_container_network }} \ {% if matrix_client_cinny_container_http_host_bind_port %} -p {{ matrix_client_cinny_container_http_host_bind_port }}:8080 \ {% endif %} + --label-file={{ matrix_client_cinny_data_path }}/labels \ --tmpfs=/tmp:rw,noexec,nosuid,size=10m \ --mount type=bind,src={{ matrix_client_cinny_data_path }}/nginx.conf,dst=/etc/nginx/nginx.conf,ro \ --mount type=bind,src={{ matrix_client_cinny_data_path }}/config.json,dst=/app/config.json,ro \ @@ -30,8 +33,15 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endfor %} {{ matrix_client_cinny_docker_image }} +{% for network in matrix_client_cinny_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-client-cinny +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-client-cinny + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-client-cinny 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-client-cinny 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-client-cinny From 6727aa55ec99f3cc9e82384b46d9111669a62ff4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian-Samuel=20Geb=C3=BChr?= Date: Tue, 14 Feb 2023 13:36:04 +0100 Subject: [PATCH 154/382] Bump element version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Julian-Samuel Gebühr --- roles/custom/matrix-client-element/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 85d8d6cf..6fd68647 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -10,7 +10,7 @@ matrix_client_element_container_image_self_build_repo: "https://github.com/vecto # - https://github.com/vector-im/element-web/issues/19544 matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}" -matrix_client_element_version: v1.11.22 +matrix_client_element_version: v1.11.23 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" From c98f40c8360d315c49ea626e12e630e7f9f5b3bd Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 14 Feb 2023 17:49:16 +0000 Subject: [PATCH 155/382] Update hydrogen 0.3.7 -> 0.3.8 --- roles/custom/matrix-client-hydrogen/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-hydrogen/defaults/main.yml b/roles/custom/matrix-client-hydrogen/defaults/main.yml index 6ead5f44..35d18cd9 100644 --- a/roles/custom/matrix-client-hydrogen/defaults/main.yml +++ b/roles/custom/matrix-client-hydrogen/defaults/main.yml @@ -6,7 +6,7 @@ matrix_client_hydrogen_enabled: true matrix_client_hydrogen_container_image_self_build: false matrix_client_hydrogen_container_image_self_build_repo: "https://github.com/vector-im/hydrogen-web.git" -matrix_client_hydrogen_version: v0.3.7 +matrix_client_hydrogen_version: v0.3.8 matrix_client_hydrogen_docker_image: "{{ matrix_client_hydrogen_docker_image_name_prefix }}vector-im/hydrogen-web:{{ matrix_client_hydrogen_version }}" matrix_client_hydrogen_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_hydrogen_container_image_self_build else 'ghcr.io/' }}" matrix_client_hydrogen_docker_image_force_pull: "{{ matrix_client_hydrogen_docker_image.endswith(':latest') }}" From 357080863315fbc69ef1280b7fa2ad546d9151ec Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 14 Feb 2023 17:50:55 +0000 Subject: [PATCH 156/382] Update synapse 1.76.0 -> 1.77.0 --- roles/custom/matrix-synapse/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 18165dd9..93d1ee02 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -36,7 +36,7 @@ matrix_synapse_container_image_customizations_dockerfile_body_custom: '' matrix_synapse_docker_image: "{{ matrix_synapse_docker_image_name_prefix }}matrixdotorg/synapse:{{ matrix_synapse_docker_image_tag }}" matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_container_image_self_build else matrix_container_global_registry_prefix }}" -matrix_synapse_version: v1.76.0 +matrix_synapse_version: v1.77.0 matrix_synapse_docker_image_tag: "{{ matrix_synapse_version }}" matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}" From 4045d72e7baafe17801fd8af8e8fd8dcc65d5cbb Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 14 Feb 2023 20:04:27 +0000 Subject: [PATCH 157/382] update postmoogle 0.9.13 -> 0.9.14 * make banlist consistent * proper multi-error message * ignore "." MX hosts * try recipient domain directly, even when MX records found, but failed --- roles/custom/matrix-bot-postmoogle/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-postmoogle/defaults/main.yml b/roles/custom/matrix-bot-postmoogle/defaults/main.yml index b3b12fef..4eeb835b 100644 --- a/roles/custom/matrix-bot-postmoogle/defaults/main.yml +++ b/roles/custom/matrix-bot-postmoogle/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_postmoogle_docker_repo: "https://gitlab.com/etke.cc/postmoogle.git" matrix_bot_postmoogle_docker_repo_version: "{{ 'main' if matrix_bot_postmoogle_version == 'latest' else matrix_bot_postmoogle_version }}" matrix_bot_postmoogle_docker_src_files_path: "{{ matrix_base_data_path }}/postmoogle/docker-src" -matrix_bot_postmoogle_version: v0.9.13 +matrix_bot_postmoogle_version: v0.9.14 matrix_bot_postmoogle_docker_image: "{{ matrix_bot_postmoogle_docker_image_name_prefix }}etke.cc/postmoogle:{{ matrix_bot_postmoogle_version }}" matrix_bot_postmoogle_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_postmoogle_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_postmoogle_docker_image_force_pull: "{{ matrix_bot_postmoogle_docker_image.endswith(':latest') }}" From c85d48c45ccc659ef7249d4e98bf0532708505bb Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Feb 2023 22:46:34 +0200 Subject: [PATCH 158/382] Remove Traefik labels for Hydrogen & Cinny from matrix-nginx-proxy Related to 6a52be79877 and 28e7ef9c71f02 --- group_vars/matrix_servers | 2 -- .../matrix-nginx-proxy/defaults/main.yml | 10 -------- .../matrix-nginx-proxy/templates/labels.j2 | 25 ------------------- 3 files changed, 37 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6e311db1..3c0737f4 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2398,8 +2398,6 @@ matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true -matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled }}" -matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index b443cc72..2b3d1eb1 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -61,16 +61,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule: "Host(`{{ matrix_ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" -matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_hostname: "{{ matrix_server_fqn_hydrogen }}" -matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_hostname }}`)" - -matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_cinny_hostname: "{{ matrix_server_fqn_cinny }}" -matrix_nginx_proxy_container_labels_traefik_proxy_cinny_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_cinny_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_cinny_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_hostname: "{{ matrix_server_fqn_buscarron }}" matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index d629ce00..2676b061 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -38,31 +38,6 @@ traefik.http.routers.matrix-nginx-proxy-matrix-federation.tls.certResolver={{ ma traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints }} {% endif %} - -{% if matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_enabled %} -# Hydrogen -traefik.http.routers.matrix-nginx-proxy-hydrogen.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_rule }} -traefik.http.routers.matrix-nginx-proxy-hydrogen.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-hydrogen.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_hydrogen_tls %} -traefik.http.routers.matrix-nginx-proxy-hydrogen.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-hydrogen.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - -{% if matrix_nginx_proxy_container_labels_traefik_proxy_cinny_enabled %} -# Cinny -traefik.http.routers.matrix-nginx-proxy-cinny.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_cinny_rule }} -traefik.http.routers.matrix-nginx-proxy-cinny.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-cinny.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_cinny_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_cinny_tls %} -traefik.http.routers.matrix-nginx-proxy-cinny.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-cinny.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - {% if matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled %} # Buscarron traefik.http.routers.matrix-nginx-proxy-buscarron.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_rule }} From 7fca8dc986c7032ad2e0c8e2097f3a46ba251297 Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Tue, 14 Feb 2023 20:38:24 -0500 Subject: [PATCH 159/382] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fa906437..174ea4c3 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ Services that run on the server to make the various parts of your installation w | [nginx](http://nginx.org/) | ✓ | Web server, listening on ports 80 and 443 - standing in front of all the other services. Using your own webserver [is possible](docs/configuring-playbook-own-webserver.md) | [Link](docs/configuring-playbook-nginx.md) | | [Let's Encrypt](https://letsencrypt.org/) | ✓ | Free SSL certificate, which secures the connection to the Synapse server and the Element web UI | [Link](docs/configuring-playbook-ssl-certificates.md) | | [ma1sd](https://github.com/ma1uta/ma1sd) | x | Matrix Identity Server | [Link](docs/configuring-playbook-ma1sd.md) -| [Exim](https://www.exim.org/) | ✓ | Mail server, through which all Matrix services send outgoing email (can be configured to relay through another SMTP server) | - | +| [Exim](https://www.exim.org/) | ✓ | Mail server, through which all Matrix services send outgoing email (can be configured to relay through another SMTP server) | [Link](docs/configuring-playbook-email.md) | | [Dimension](https://github.com/turt2live/matrix-dimension) | x | An open source integrations manager for matrix clients | [Link](docs/configuring-playbook-dimension.md) | | [Sygnal](https://github.com/matrix-org/sygnal) | x | Push gateway | [Link](docs/configuring-playbook-sygnal.md) | | [ntfy](https://ntfy.sh) | x | Push notifications server | [Link](docs/configuring-playbook-ntfy.md) | From 0b9dc56edffb3f85deb2919a7d4e2f9f69c0fe78 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 06:03:55 +0200 Subject: [PATCH 160/382] Add type support to matrix_coturn_container_additional_volumes .. and try to auto-switch between `bind` and `volume` depending on whether there's a slash in the `src` path. Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2482 --- roles/custom/matrix-coturn/defaults/main.yml | 3 ++- .../matrix-coturn/templates/systemd/matrix-coturn.service.j2 | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 47577a46..9922f1d0 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -39,7 +39,8 @@ matrix_coturn_systemd_required_services_list: ['docker.service'] # A list of additional "volumes" to mount in the container. # This list gets populated dynamically at runtime. You can provide a different default value, # if you wish to mount your own files into the container. -# Contains definition objects like this: `{"src": "/outside", "dst": "/inside", "options": "rw|ro|slave|.."} +# Contains definition objects like this: `{"type": "bind", "src": "/outside", "dst": "/inside", "options": "readonly"}. +# See the `--mount` documentation for the `docker run` command. matrix_coturn_container_additional_volumes: [] # A list of extra arguments to pass to the container diff --git a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 index 523ad1cc..2c0c9cd9 100644 --- a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 +++ b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 @@ -35,7 +35,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endif %} --mount type=bind,src={{ matrix_coturn_config_path }},dst=/turnserver.conf,ro \ {% for volume in matrix_coturn_container_additional_volumes %} - --mount type=bind,src={{ volume.src }},dst={{ volume.dst }}{{ (',' + volume.options) if volume.options else '' }} \ + --mount type={{ volume.type | default('bind' if '/' in volume.src else 'volume') }},src={{ volume.src }},dst={{ volume.dst }}{{ (',' + volume.options) if volume.options else '' }} \ {% endfor %} {% for arg in matrix_coturn_container_extra_arguments %} {{ arg }} \ From b648d4579f65d0cdd042d22124ff20ed3bc7323e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 07:30:36 +0200 Subject: [PATCH 161/382] Make it clear that certain matrix_nginx_proxy_* variables are necessary for now, even with Traefik Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2480 --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b2716ef..bbf1772d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -74,7 +74,7 @@ Unless we have some regression, **existing users should be able to update their #### How do I explicitly switch to Traefik right now? -**Users who want to migrate to Traefik** today, can do so by using configuration like this: +**Users who wish to migrate to Traefik** today, can do so by **adding** this to their configuration: ```yaml matrix_playbook_reverse_proxy_type: playbook-managed-traefik @@ -82,6 +82,8 @@ matrix_playbook_reverse_proxy_type: playbook-managed-traefik devture_traefik_ssl_email_address: YOUR_EMAIL_ADDRESS ``` +You may still need to keep certain old `matrix_nginx_proxy_*` variables (like `matrix_nginx_proxy_base_domain_serving_enabled`), even when using Traefik. For now, we recommend keeping all `matrix_nginx_proxy_*` variables just in case. In the future, reliance on `matrix-nginx-proxy` will be removed. + Switching to Traefik will obtain new SSL certificates from Let's Encrypt (stored in `/devture-traefik/ssl/acme.json`). **The switch is reversible**. You can always go back to `playbook-managed-nginx` if Traefik is causing you trouble. **Note**: toggling `matrix_playbook_reverse_proxy_type` between Traefik and nginx will uninstall the Traefik role and all of its data (under `/devture-traefik`), so you may run into a Let's Encrypt rate limit if you do it often. From b6d45304b4b19639539b9e31f6589009d54fe0b1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 08:49:30 +0200 Subject: [PATCH 162/382] Make sure group_vars/matrix_servers is indented like other YAML files (with 2 spaces) --- .editorconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.editorconfig b/.editorconfig index 6d5dc09a..996cf9b4 100644 --- a/.editorconfig +++ b/.editorconfig @@ -15,7 +15,7 @@ trim_trailing_whitespace = true ######################### # YAML Files -[*.{yml,yaml,log.config.j2,yaml.j2}] +[*.{yml,yaml,log.config.j2,yaml.j2,group_vars/matrix_servers}] indent_style = space indent_size = 2 From 59a3646c66dcc0314c9622907c5278ec10e05e30 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 08:50:31 +0200 Subject: [PATCH 163/382] Make .editorconfig for group_vars/matrix_servers cleaner It's the same result as b6d45304b4b19, but it's done in a cleaner way. --- .editorconfig | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.editorconfig b/.editorconfig index 996cf9b4..fac22d42 100644 --- a/.editorconfig +++ b/.editorconfig @@ -15,7 +15,11 @@ trim_trailing_whitespace = true ######################### # YAML Files -[*.{yml,yaml,log.config.j2,yaml.j2,group_vars/matrix_servers}] +[*.{yml,yaml,log.config.j2,yaml.j2}] +indent_style = space +indent_size = 2 + +[group_vars/matrix_servers] indent_style = space indent_size = 2 From 94124263a7c3414b96fdd72cc35dbf16f7af57d6 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 08:56:03 +0200 Subject: [PATCH 164/382] Add matrix_prometheus_container_network/matrix_prometheus_container_additional_networks --- group_vars/matrix_servers | 2 ++ roles/custom/matrix-prometheus/defaults/main.yml | 8 ++++++++ .../matrix-prometheus/tasks/setup_install.yml | 5 +++++ .../matrix-prometheus/tasks/validate_config.yml | 8 ++++++++ .../templates/systemd/matrix-prometheus.service.j2 | 14 +++++++++++--- 5 files changed, 34 insertions(+), 3 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 3c0737f4..3e64c14a 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3296,6 +3296,8 @@ matrix_prometheus_nginxlog_exporter_enabled: false matrix_prometheus_enabled: false +matrix_prometheus_container_network: "{{ matrix_docker_network }}" + # Normally, matrix-nginx-proxy is enabled and nginx can reach Prometheus over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Prometheus' HTTP port to the local host. diff --git a/roles/custom/matrix-prometheus/defaults/main.yml b/roles/custom/matrix-prometheus/defaults/main.yml index 956faf46..fddacfb4 100644 --- a/roles/custom/matrix-prometheus/defaults/main.yml +++ b/roles/custom/matrix-prometheus/defaults/main.yml @@ -22,6 +22,14 @@ matrix_prometheus_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-prometheus.service wants matrix_prometheus_systemd_wanted_services_list: [] +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_prometheus_container_network: '' + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_prometheus_container_additional_networks: [] + # Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9090 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:9090"), or empty string to not expose. diff --git a/roles/custom/matrix-prometheus/tasks/setup_install.yml b/roles/custom/matrix-prometheus/tasks/setup_install.yml index 1d3d6669..f606e268 100644 --- a/roles/custom/matrix-prometheus/tasks/setup_install.yml +++ b/roles/custom/matrix-prometheus/tasks/setup_install.yml @@ -45,6 +45,11 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" +- name: Ensure Prometheus container network is created + community.general.docker_network: + name: "{{ matrix_prometheus_container_network }}" + driver: bridge + - name: Ensure matrix-prometheus.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-prometheus.service.j2" diff --git a/roles/custom/matrix-prometheus/tasks/validate_config.yml b/roles/custom/matrix-prometheus/tasks/validate_config.yml index 3e386304..49121e7e 100644 --- a/roles/custom/matrix-prometheus/tasks/validate_config.yml +++ b/roles/custom/matrix-prometheus/tasks/validate_config.yml @@ -5,3 +5,11 @@ msg: > You need to enable `matrix_prometheus_scraper_synapse_enabled` and/or `matrix_prometheus_scraper_node_enabled` for Prometheus grab metrics. when: "not matrix_prometheus_scraper_synapse_enabled and not matrix_prometheus_scraper_node_enabled" + +- name: Fail if required Prometheus settings not defined + ansible.builtin.fail: + msg: > + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_prometheus_container_network diff --git a/roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 b/roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 index 58455734..2b716db6 100644 --- a/roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 +++ b/roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 @@ -16,13 +16,14 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus 2>/dev/null || true' - -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-prometheus \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-prometheus \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_prometheus_container_network }} \ {% if matrix_prometheus_container_http_host_bind_port %} -p {{ matrix_prometheus_container_http_host_bind_port }}:9090 \ {% endif %} @@ -33,8 +34,15 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endfor %} {{ matrix_prometheus_docker_image }} {{ matrix_prometheus_process_arguments|join(' ') }} +{% for network in matrix_prometheus_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-prometheus +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-prometheus + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-prometheus From 1006b8d8996b81c7f44914bf86ddef9a2653cdde Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 10:30:58 +0200 Subject: [PATCH 165/382] Replace matrix-grafana with an external role --- CHANGELOG.md | 11 +++ ...configuring-playbook-prometheus-grafana.md | 14 ++-- ...onfiguring-playbook-prometheus-nginxlog.md | 6 +- group_vars/matrix_servers | 69 +++++++++++++----- playbooks/matrix.yml | 2 +- requirements.yml | 3 + roles/custom/matrix-grafana/defaults/main.yml | 65 ----------------- roles/custom/matrix-grafana/tasks/main.yml | 20 ----- .../matrix-grafana/tasks/setup_install.yml | 73 ------------------- .../matrix-grafana/tasks/setup_uninstall.yml | 20 ----- .../matrix-grafana/tasks/validate_config.yml | 16 ---- .../templates/dashboards.yaml.j2 | 9 --- .../templates/datasources.yaml.j2 | 8 -- .../matrix-grafana/templates/grafana.ini.j2 | 29 -------- .../systemd/matrix-grafana.service.j2 | 43 ----------- .../matrix-nginx-proxy/defaults/main.yml | 5 -- .../matrix-nginx-proxy/templates/labels.j2 | 13 ---- .../tasks/setup_install.yml | 4 +- .../tasks/setup_uninstall.yml | 2 +- .../defaults/main.yml | 3 + .../tasks/validate_config.yml | 15 ++++ 21 files changed, 97 insertions(+), 333 deletions(-) delete mode 100644 roles/custom/matrix-grafana/defaults/main.yml delete mode 100644 roles/custom/matrix-grafana/tasks/main.yml delete mode 100644 roles/custom/matrix-grafana/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-grafana/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-grafana/tasks/validate_config.yml delete mode 100644 roles/custom/matrix-grafana/templates/dashboards.yaml.j2 delete mode 100644 roles/custom/matrix-grafana/templates/datasources.yaml.j2 delete mode 100644 roles/custom/matrix-grafana/templates/grafana.ini.j2 delete mode 100644 roles/custom/matrix-grafana/templates/systemd/matrix-grafana.service.j2 diff --git a/CHANGELOG.md b/CHANGELOG.md index bbf1772d..b47433d7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,14 @@ +# 2023-02-15 + +## The matrix-grafana role lives independently now + +**TLDR**: the `matrix-grafana` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +The `matrix-grafana` role (which configures [Grafana](docs/configuring-playbook-prometheus-grafana.md)) has been extracted from the playbook and now lives in its [own repository](https://gitlab.com/etke.cc/roles/grafana). This makes it possible to easily use it in other Ansible playbooks. + +You need to **update you roles** (`just roles` or `make roles`) regardless of whether you're enabling Grafana or not. If you're making use of Grafana via this playbook, you will need to update variable references in your `vars.yml` file (`matrix_grafana_` -> `grafana_`). + + # 2023-02-13 ## The matrix-backup-borg role lives independently now diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index 956b0573..de05d690 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -18,16 +18,16 @@ prometheus_postgres_exporter_enabled: true # You can remove this, if unnecessary. matrix_prometheus_nginxlog_exporter_enabled: true -matrix_grafana_enabled: true +grafana_enabled: true -matrix_grafana_anonymous_access: false +grafana_anonymous_access: false # This has no relation to your Matrix user id. It can be any username you'd like. # Changing the username subsequently won't work. -matrix_grafana_default_admin_user: "some_username_chosen_by_you" +grafana_default_admin_user: "some_username_chosen_by_you" # Changing the password subsequently won't work. -matrix_grafana_default_admin_password: "some_strong_password_chosen_by_you" +grafana_default_admin_password: "some_strong_password_chosen_by_you" ``` By default, a [Grafana](https://grafana.com/) web user-interface will be available at `https://stats.`. @@ -43,9 +43,9 @@ Name | Description `prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures `prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. `matrix_prometheus_nginxlog_exporter_enabled`|[NGINX Log Exporter](configuring-playbook-prometheus-nginxlog.md) is an addon of sorts to expose NGINX logs to Prometheus. -`matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in -`matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option. -`matrix_grafana_default_admin_user`
`matrix_grafana_default_admin_password`|By default Grafana creates a user with `admin` as the username and password. If you feel this is insecure and you want to change it beforehand, you can do that here +`grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in +`grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option. +`grafana_default_admin_user`
`grafana_default_admin_password`|By default Grafana creates a user with `admin` as the username and password. If you feel this is insecure and you want to change it beforehand, you can do that here ## Security and privacy diff --git a/docs/configuring-playbook-prometheus-nginxlog.md b/docs/configuring-playbook-prometheus-nginxlog.md index 832efbd8..321a1d6e 100644 --- a/docs/configuring-playbook-prometheus-nginxlog.md +++ b/docs/configuring-playbook-prometheus-nginxlog.md @@ -15,13 +15,13 @@ matrix_prometheus_nginxlog_exporter_enabled: true matrix_prometheus_enabled: true # optional for visualization -matrix_grafana_enabled: true +grafana_enabled: true ``` x | Prerequisites | Variable | Description |:--:|:--:|:--:|:--| **REQUIRED** | `matrix-prometheus`| `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. -_Optional_ | [`matrix-grafana`](configuring-playbook-prometheus-grafana.md) | [`matrix_grafana_enabled`](configuring-playbook-prometheus-grafana.md)|[Grafana](https://grafana.com) is the visual component. It shows (on the `stats.` subdomain) graphs that we're interested in. When enabled the `NGINX PROXY` dashboard is automatically added. +_Optional_ | [`matrix-grafana`](configuring-playbook-prometheus-grafana.md) | [`grafana_enabled`](configuring-playbook-prometheus-grafana.md)|[Grafana](https://grafana.com) is the visual component. It shows (on the `stats.` subdomain) graphs that we're interested in. When enabled the `NGINX PROXY` dashboard is automatically added. ## Docker Image Compatibility @@ -56,4 +56,4 @@ The following variables may be of interest: Name | Description -----|---------- -`matrix_nginx_proxy_proxy_matrix_metrics_enabled`|Set this to `true` to enable metrics exposure for various services on `https://matrix.DOMAIN/metrics/*`. Refer to the individual `matrix_SERVICE_metrics_proxying_enabled` variables below for exposing metrics for each individual service. \ No newline at end of file +`matrix_nginx_proxy_proxy_matrix_metrics_enabled`|Set this to `true` to enable metrics exposure for various services on `https://matrix.DOMAIN/metrics/*`. Refer to the individual `matrix_SERVICE_metrics_proxying_enabled` variables below for exposing metrics for each individual service. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 3e64c14a..76b718ef 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -306,7 +306,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-etherpad.service', 'priority': 4000, 'groups': ['matrix', 'etherpad']}] if matrix_etherpad_enabled else []) + - ([{'name': 'matrix-grafana.service', 'priority': 4000, 'groups': ['matrix', 'monitoring', 'grafana']}] if matrix_grafana_enabled else []) + ([{'name': (grafana_identifier + '.service'), 'priority': 4000, 'groups': ['matrix', 'monitoring', 'grafana']}] if grafana_enabled else []) + ([{'name': 'matrix-jitsi-web.service', 'priority': 4200, 'groups': ['matrix', 'jitsi', 'jitsi-web']}] if matrix_jitsi_enabled else []) + @@ -2389,7 +2389,7 @@ matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" matrix_nginx_proxy_proxy_jitsi_manage_wellknown: "{{ matrix_jitsi_require_well_known }}" -matrix_nginx_proxy_proxy_grafana_enabled: "{{ matrix_grafana_enabled }}" +matrix_nginx_proxy_proxy_grafana_enabled: "{{ grafana_enabled }}" matrix_nginx_proxy_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" matrix_nginx_proxy_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" @@ -2403,7 +2403,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" -matrix_nginx_proxy_container_labels_traefik_proxy_grafana_enabled: "{{ matrix_grafana_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" @@ -2481,7 +2480,7 @@ matrix_nginx_proxy_systemd_wanted_services_list: | + (['matrix-client-hydrogen.service'] if matrix_client_hydrogen_enabled else []) + - (['matrix-grafana.service'] if matrix_grafana_enabled else []) + ([(grafana_idenifier + '.service')] if grafana_enabled else []) + (['matrix-dimension.service'] if matrix_dimension_enabled else []) + @@ -2520,7 +2519,7 @@ matrix_ssl_domains_to_obtain_certificates_for: | + ([matrix_server_fqn_jitsi] if matrix_jitsi_enabled else []) + - ([matrix_server_fqn_grafana] if matrix_grafana_enabled else []) + ([matrix_server_fqn_grafana] if grafana_enabled else []) + ([matrix_server_fqn_sygnal] if matrix_sygnal_enabled else []) + @@ -2938,7 +2937,7 @@ matrix_client_element_container_additional_networks: "{{ [matrix_playbook_revers matrix_client_element_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_client_element_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" -matrix_client_element_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" +matrix_client_element_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_client_element_default_hs_url: "{{ matrix_homeserver_url }}" matrix_client_element_default_is_url: "{{ matrix_identity_server_url }}" @@ -2991,7 +2990,7 @@ matrix_client_hydrogen_container_additional_networks: "{{ [matrix_playbook_rever matrix_client_hydrogen_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_client_hydrogen_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" -matrix_client_hydrogen_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" +matrix_client_hydrogen_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_client_hydrogen_default_hs_url: "{{ matrix_homeserver_url }}" @@ -3024,7 +3023,7 @@ matrix_client_cinny_container_additional_networks: "{{ [matrix_playbook_reverse_ matrix_client_cinny_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_client_cinny_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" -matrix_client_cinny_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" +matrix_client_cinny_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_client_cinny_default_hs_url: "{{ matrix_homeserver_url }}" @@ -3192,7 +3191,7 @@ matrix_synapse_admin_container_additional_networks: "{{ [matrix_playbook_reverse matrix_synapse_admin_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_synapse_admin_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" -matrix_synapse_admin_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary if devture_traefik_enabled else '' }}" +matrix_synapse_admin_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" ###################################################################### # @@ -3329,18 +3328,52 @@ matrix_prometheus_scraper_nginxlog_server_port: "{{ (matrix_prometheus_nginxlog_ ###################################################################### # -# matrix-grafana +# etke/grafana # ###################################################################### -matrix_grafana_enabled: false +grafana_enabled: false + +grafana_identifier: matrix-grafana + +grafana_uid: "{{ matrix_user_uid }}" +grafana_gid: "{{ matrix_user_gid }}" + +grafana_hostname: "{{ matrix_server_fqn_grafana }}" + +grafana_base_path: "{{ matrix_base_data_path }}/grafana" + +grafana_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else grafana_identifier }}" + +grafana_container_additional_networks: | + {{ + ( + ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + + + ([matrix_prometheus_container_network] if matrix_prometheus_enabled and matrix_prometheus_container_network != grafana_container_network else []) + ) | unique + }} + +grafana_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +grafana_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +grafana_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" # Normally, matrix-nginx-proxy is enabled and nginx can reach Grafana over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Grafana's HTTP port to the local host. -matrix_grafana_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '3000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +grafana_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '3000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" + +grafana_provisioning_datasources: | + {{ + ([{ + 'name': (matrix_server_fqn_matrix + ' - Prometheus'), + 'type': 'prometheus', + 'access': 'proxy', + 'url': 'http://matrix-prometheus:9090', + }] if matrix_prometheus_enabled else []) + }} -matrix_grafana_dashboard_download_urls: | +grafana_dashboard_download_urls: | {{ (matrix_synapse_grafana_dashboard_urls if matrix_homeserver_implementation == 'synapse' and matrix_synapse_metrics_enabled else []) + @@ -3351,18 +3384,18 @@ matrix_grafana_dashboard_download_urls: | (matrix_prometheus_nginxlog_exporter_dashboard_urls if matrix_prometheus_nginxlog_exporter_enabled else []) }} -matrix_grafana_default_home_dashboard_path: |- +grafana_default_home_dashboard_path: |- {{ { - 'synapse': ('/etc/grafana/dashboards/synapse.json' if matrix_synapse_metrics_enabled else '/etc/grafana/dashboards/node-exporter-full.json'), - 'dendrite': '/etc/grafana/dashboards/node-exporter-full.json', - 'conduit': '/etc/grafana/dashboards/node-exporter-full.json', + 'synapse': ('/etc/grafana/dashboards/synapse.json' if matrix_synapse_metrics_enabled and matrix_synapse_metrics_enabled else ('/etc/grafana/dashboards/node-exporter-full.json' if prometheus_node_exporter_enabled else '')), + 'dendrite': ('/etc/grafana/dashboards/node-exporter-full.json' if prometheus_node_exporter_enabled else ''), + 'conduit': ('/etc/grafana/dashboards/node-exporter-full.json' if prometheus_node_exporter_enabled else ''), }[matrix_homeserver_implementation] }} ###################################################################### # -# /matrix-grafana +# /etke/grafana # ###################################################################### diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index d976a012..f5e21493 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -91,7 +91,7 @@ - galaxy/prometheus_postgres_exporter - custom/matrix-prometheus-nginxlog-exporter - custom/matrix-prometheus - - custom/matrix-grafana + - galaxy/grafana - custom/matrix-prometheus-services-proxy-connect - custom/matrix-registration - custom/matrix-client-element diff --git a/requirements.yml b/requirements.yml index 1244c465..829ef99b 100644 --- a/requirements.yml +++ b/requirements.yml @@ -39,6 +39,9 @@ - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git version: v1.2.3-1.7.6-0 +- src: git+https://gitlab.com/etke.cc/roles/grafana.git + version: v9.3.6-0 + - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: fb09fd26f877372417d5586f1e79e83f983f0bd6 diff --git a/roles/custom/matrix-grafana/defaults/main.yml b/roles/custom/matrix-grafana/defaults/main.yml deleted file mode 100644 index 5b730480..00000000 --- a/roles/custom/matrix-grafana/defaults/main.yml +++ /dev/null @@ -1,65 +0,0 @@ ---- -# matrix-grafana is open source visualization and analytics software -# See: https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md -# Project source code URL: https://github.com/grafana/grafana - -matrix_grafana_enabled: true - -matrix_grafana_version: 9.3.6 -matrix_grafana_docker_image: "{{ matrix_container_global_registry_prefix }}grafana/grafana:{{ matrix_grafana_version }}" -matrix_grafana_docker_image_force_pull: "{{ matrix_grafana_docker_image.endswith(':latest') }}" - -# matrix_grafana_dashboard_download_urls holds a list of URLs of dashboards to download -matrix_grafana_dashboard_download_urls: [] - -matrix_grafana_base_path: "{{ matrix_base_data_path }}/grafana" -matrix_grafana_config_path: "{{ matrix_grafana_base_path }}/config" -matrix_grafana_data_path: "{{ matrix_grafana_base_path }}/data" - -# Allow viewing Grafana without logging in -matrix_grafana_anonymous_access: false - -# When `false`, sends a `X-Frame-Options: deny` HTTP header, which allows Grafana from being embeded in a frame. -# Read more here: https://grafana.com/docs/grafana/latest/administration/configuration/#allow_embedding -matrix_grafana_allow_embedding: false - -# specify organization name that should be used for unauthenticated users -# if you change this in the Grafana admin panel, this needs to be updated -# to match to keep anonymous logins working -matrix_grafana_anonymous_access_org_name: 'Main Org.' - - -# default admin credentials, you are asked to change these on first login -matrix_grafana_default_admin_user: admin -matrix_grafana_default_admin_password: admin - -# Set to true to add the Content-Security-Policy header to your requests. -# CSP allows to control resources that the user agent can load and helps -# prevent XSS attacks. -# [Content Security Policy](https://grafana.com/docs/grafana/latest/administration/configuration/#content_security_policy) -matrix_grafana_content_security_policy: true - -# specify content security policy template to customized template -# added https: and http: url schemes (ignored by browsers supporting 'strict-dynamic') to be backward compatible with older browsers. -# [Content Security Policy Browser Test] (https://content-security-policy.com/browser-test/) -# [Content Security Policy Reference](https://content-security-policy.com/script-src/) -matrix_grafana_content_security_policy_customized: false -matrix_grafana_content_security_policy_template: "script-src 'self' 'unsafe-eval' 'unsafe-inline' http: https: 'strict-dynamic' $NONCE;object-src 'none';font-src 'self';style-src 'self' 'unsafe-inline' blob:;img-src * data:;base-uri 'self';connect-src 'self' grafana.com ws://$ROOT_PATH wss://$ROOT_PATH;manifest-src 'self';media-src 'none';form-action 'self';" - -# matrix_grafana_default_home_dashboard_path influences the `default_home_dashboard_path` grafana.ini setting, -# which is an in-container path for the default dashboard. -matrix_grafana_default_home_dashboard_path: /etc/grafana/dashboards/node-exporter-full.json - -# A list of extra arguments to pass to the container -matrix_grafana_container_extra_arguments: [] - -# List of systemd services that matrix-grafana.service depends on -matrix_grafana_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-grafana.service wants -matrix_grafana_systemd_wanted_services_list: [] - -# Controls whether the matrix-grafana container exposes its HTTP port (tcp/3000 in the container). -# -# Takes an ":" or "" value (e.g. "127.0.0.1:3000"), or empty string to not expose. -matrix_grafana_container_http_host_bind_port: '' diff --git a/roles/custom/matrix-grafana/tasks/main.yml b/roles/custom/matrix-grafana/tasks/main.yml deleted file mode 100644 index 263d894f..00000000 --- a/roles/custom/matrix-grafana/tasks/main.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- block: - - when: matrix_grafana_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - - - when: matrix_grafana_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-grafana - - install-all - - install-grafana - -- block: - - when: not matrix_grafana_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-grafana diff --git a/roles/custom/matrix-grafana/tasks/setup_install.yml b/roles/custom/matrix-grafana/tasks/setup_install.yml deleted file mode 100644 index 86a6b04b..00000000 --- a/roles/custom/matrix-grafana/tasks/setup_install.yml +++ /dev/null @@ -1,73 +0,0 @@ ---- - -- name: Ensure matrix-grafana image is pulled - community.docker.docker_image: - name: "{{ matrix_grafana_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_grafana_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_grafana_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure grafana paths exists - ansible.builtin.file: - path: "{{ item }}" - state: directory - mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - with_items: - - "{{ matrix_grafana_base_path }}" - - "{{ matrix_grafana_config_path }}" - - "{{ matrix_grafana_config_path }}/provisioning" - - "{{ matrix_grafana_config_path }}/provisioning/datasources" - - "{{ matrix_grafana_config_path }}/provisioning/dashboards" - - "{{ matrix_grafana_config_path }}/dashboards" - - "{{ matrix_grafana_data_path }}" - -- name: Ensure grafana.ini present - ansible.builtin.template: - src: "{{ role_path }}/templates/grafana.ini.j2" - dest: "{{ matrix_grafana_config_path }}/grafana.ini" - mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - -- name: Ensure provisioning/datasources/default.yaml present - ansible.builtin.template: - src: "{{ role_path }}/templates/datasources.yaml.j2" - dest: "{{ matrix_grafana_config_path }}/provisioning/datasources/default.yaml" - mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - -- name: Ensure provisioning/dashboards/default.yaml present - ansible.builtin.template: - src: "{{ role_path }}/templates/dashboards.yaml.j2" - dest: "{{ matrix_grafana_config_path }}/provisioning/dashboards/default.yaml" - mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - -- name: Ensure dashboard(s) downloaded - ansible.builtin.get_url: - url: "{{ item }}" - dest: "{{ matrix_grafana_config_path }}/dashboards/" - force: true - mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - with_items: "{{ matrix_grafana_dashboard_download_urls }}" - register: result - retries: "{{ devture_playbook_help_geturl_retries_count }}" - delay: "{{ devture_playbook_help_geturl_retries_delay }}" - until: result is not failed - -- name: Ensure matrix-grafana.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-grafana.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-grafana.service" - mode: 0644 - register: matrix_grafana_systemd_service_result diff --git a/roles/custom/matrix-grafana/tasks/setup_uninstall.yml b/roles/custom/matrix-grafana/tasks/setup_uninstall.yml deleted file mode 100644 index 12c3cde9..00000000 --- a/roles/custom/matrix-grafana/tasks/setup_uninstall.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- name: Check existence of matrix-grafana service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-grafana.service" - register: matrix_grafana_service_stat - -- when: matrix_grafana_service_stat.stat.exists | bool - block: - - name: Ensure matrix-grafana is stopped - ansible.builtin.service: - name: matrix-grafana - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-grafana.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-grafana.service" - state: absent diff --git a/roles/custom/matrix-grafana/tasks/validate_config.yml b/roles/custom/matrix-grafana/tasks/validate_config.yml deleted file mode 100644 index 21c44dc8..00000000 --- a/roles/custom/matrix-grafana/tasks/validate_config.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- - -- name: Fail if Prometheus not enabled - ansible.builtin.fail: - msg: > - You need to enable `matrix_prometheus_enabled` to use Prometheus as data source for Grafana. - when: "not matrix_prometheus_enabled" - -- name: (Deprecation) Catch and report renamed settings - ansible.builtin.fail: - msg: >- - Your configuration contains a variable, which now has a different name. - Please change your configuration to rename the variable (`{{ item.old }}` -> `{{ item.new }}`). - when: "item.old in vars" - with_items: - - {'old': 'matrix_grafana_dashboard_download_urls_all', 'new': 'matrix_grafana_dashboard_download_urls'} diff --git a/roles/custom/matrix-grafana/templates/dashboards.yaml.j2 b/roles/custom/matrix-grafana/templates/dashboards.yaml.j2 deleted file mode 100644 index aae42ba2..00000000 --- a/roles/custom/matrix-grafana/templates/dashboards.yaml.j2 +++ /dev/null @@ -1,9 +0,0 @@ -apiVersion: 1 - -providers: - - name: {{ matrix_server_fqn_matrix }} - Dashboards - folder: '' # The folder where to place the dashboards - type: file - allowUiUpdates: true - options: - path: /etc/grafana/dashboards diff --git a/roles/custom/matrix-grafana/templates/datasources.yaml.j2 b/roles/custom/matrix-grafana/templates/datasources.yaml.j2 deleted file mode 100644 index 6ccbe374..00000000 --- a/roles/custom/matrix-grafana/templates/datasources.yaml.j2 +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: 1 - -datasources: - - name: {{ matrix_server_fqn_matrix }} - Prometheus - type: prometheus - # Access mode - proxy (server in the UI) or direct (browser in the UI). - access: proxy - url: http://matrix-prometheus:9090 diff --git a/roles/custom/matrix-grafana/templates/grafana.ini.j2 b/roles/custom/matrix-grafana/templates/grafana.ini.j2 deleted file mode 100644 index ac609f91..00000000 --- a/roles/custom/matrix-grafana/templates/grafana.ini.j2 +++ /dev/null @@ -1,29 +0,0 @@ -[server] -root_url = "https://{{ matrix_server_fqn_grafana }}" - -[security] -# default admin user, created on startup -admin_user = "{{ matrix_grafana_default_admin_user }}" - -# default admin password, can be changed before first start of grafana, or in profile settings -admin_password = """{{ matrix_grafana_default_admin_password }}""" - -# specify content_security_policy to add the Content-Security-Policy header to your requests -content_security_policy = "{{ matrix_grafana_content_security_policy }}" - -# specify content security policy template to customized template -{% if matrix_grafana_content_security_policy_customized %} -content_security_policy_template = """{{ matrix_grafana_content_security_policy_template }}""" -{% endif %} - -allow_embedding = {{ matrix_grafana_allow_embedding }} - -[auth.anonymous] -# enable anonymous access -enabled = {{ matrix_grafana_anonymous_access }} - -# specify organization name that should be used for unauthenticated users -org_name = "{{ matrix_grafana_anonymous_access_org_name }}" - -[dashboards] -default_home_dashboard_path = {{ matrix_grafana_default_home_dashboard_path }} diff --git a/roles/custom/matrix-grafana/templates/systemd/matrix-grafana.service.j2 b/roles/custom/matrix-grafana/templates/systemd/matrix-grafana.service.j2 deleted file mode 100644 index fd48b01e..00000000 --- a/roles/custom/matrix-grafana/templates/systemd/matrix-grafana.service.j2 +++ /dev/null @@ -1,43 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=matrix-grafana -{% for service in matrix_grafana_systemd_required_services_list %} -Requires={{ service }} -After={{ service }} -{% endfor %} -{% for service in matrix_grafana_systemd_wanted_services_list %} -Wants={{ service }} -{% endfor %} -DefaultDependencies=no - -[Service] -Type=simple -Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-grafana 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-grafana 2>/dev/null || true' - - -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-grafana \ - --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --cap-drop=ALL \ - --read-only \ - --network={{ matrix_docker_network }} \ - {% if matrix_grafana_container_http_host_bind_port %} - -p {{ matrix_grafana_container_http_host_bind_port }}:3000 \ - {% endif %} - -v {{ matrix_grafana_config_path }}:/etc/grafana:z \ - -v {{ matrix_grafana_data_path }}:/var/lib/grafana:z \ - {% for arg in matrix_grafana_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - {{ matrix_grafana_docker_image }} - -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-grafana 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-grafana 2>/dev/null || true' -Restart=always -RestartSec=30 -SyslogIdentifier=matrix-grafana - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 2b3d1eb1..38afb3c4 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -86,11 +86,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_hostname: "{{ matrix_ser matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_hostname }}`)" -matrix_nginx_proxy_container_labels_traefik_proxy_grafana_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_grafana_hostname: "{{ matrix_server_fqn_grafana }}" -matrix_nginx_proxy_container_labels_traefik_proxy_grafana_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_grafana_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_grafana_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}" matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 2676b061..4565da19 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -97,19 +97,6 @@ traefik.http.routers.matrix-nginx-proxy-jitsi.tls.certResolver={{ matrix_nginx_p traefik.http.routers.matrix-nginx-proxy-jitsi.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} {% endif %} - -{% if matrix_nginx_proxy_container_labels_traefik_proxy_grafana_enabled %} -# Grafana -traefik.http.routers.matrix-nginx-proxy-grafana.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_grafana_rule }} -traefik.http.routers.matrix-nginx-proxy-grafana.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-grafana.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_grafana_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_grafana_tls %} -traefik.http.routers.matrix-nginx-proxy-grafana.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-grafana.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - {% if matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled %} # Sygnal traefik.http.routers.matrix-nginx-proxy-sygnal.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_rule }} diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml index 9b2ac3e6..2835b2c9 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml @@ -33,11 +33,11 @@ - name: Ensure grafana dashboard is installed ansible.builtin.template: src: "{{ role_path }}/templates/nginx-proxy.json" - dest: "{{ matrix_grafana_config_path }}/dashboards/" + dest: "{{ grafana_config_path }}/dashboards/" mode: 0644 owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" - when: matrix_grafana_enabled + when: grafana_enabled - name: Ensure matrix-prometheus-nginxlog-exporter.service installed ansible.builtin.template: diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml index c0d0a833..6fce44ae 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml @@ -35,5 +35,5 @@ - name: Ensure matrix-prometheus-nginxlog-exporter grafana dashboard doesn't exist ansible.builtin.file: - path: "{{ matrix_grafana_config_path }}/dashboards/nginx-proxy.json" + path: "{{ grafana_config_path }}/dashboards/nginx-proxy.json" state: absent diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml index 109ecd72..03867679 100644 --- a/roles/custom/matrix_playbook_migration/defaults/main.yml +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -8,3 +8,6 @@ matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_validati # Controls if (`matrix_backup_borg` -> `backup_borg`) validation will run. matrix_playbook_migration_matrix_backup_borg_migration_validation_enabled: true + +# Controls if (`matrix_grafana` -> `grafana`) validation will run. +matrix_playbook_migration_matrix_grafana_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index f588ed08..2ac04ece 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -95,3 +95,18 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_backup_borg_` -> `backup_borg_`). We found usage of the following variables: {{ matrix_playbook_migration_backup_borg_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_backup_borg_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_grafana_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_grafana_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_grafana_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_grafana variables + ansible.builtin.fail: + msg: >- + The matrix-grafana role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/grafana. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_grafana_` -> `grafana_`). + We found usage of the following variables: {{ matrix_playbook_migration_grafana_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_grafana_migration_vars | length > 0" From 7c5826f1c334d1458d010ee0bb2af3a768584367 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 10:52:25 +0200 Subject: [PATCH 166/382] Break dependency between matrix-prometheus-nginxlog-exporter and the Grafana role Wiring happens via `group_vars/matrix_servers` now. --- group_vars/matrix_servers | 8 + requirements.yml | 2 +- .../tasks/setup_install.yml | 10 - .../templates/grafana/nginx-proxy.json | 1705 +++++++++++++++++ 4 files changed, 1714 insertions(+), 11 deletions(-) create mode 100644 roles/custom/matrix-prometheus-nginxlog-exporter/templates/grafana/nginx-proxy.json diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 76b718ef..78e15eca 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3384,6 +3384,14 @@ grafana_dashboard_download_urls: | (matrix_prometheus_nginxlog_exporter_dashboard_urls if matrix_prometheus_nginxlog_exporter_enabled else []) }} +grafana_provisioning_dashboard_template_files: | + {{ + ([{ + 'path': 'roles/custom/matrix-prometheus-nginxlog-exporter/templates/grafana/nginx-proxy.json', + 'name': 'nginx-proxy.json', + }] if matrix_prometheus_nginxlog_exporter_enabled else []) + }} + grafana_default_home_dashboard_path: |- {{ { diff --git a/requirements.yml b/requirements.yml index 829ef99b..716b7c6e 100644 --- a/requirements.yml +++ b/requirements.yml @@ -40,7 +40,7 @@ version: v1.2.3-1.7.6-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git - version: v9.3.6-0 + version: v9.3.6-1 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: fb09fd26f877372417d5586f1e79e83f983f0bd6 diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml index 2835b2c9..ca3bf3a8 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml @@ -30,18 +30,8 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" -- name: Ensure grafana dashboard is installed - ansible.builtin.template: - src: "{{ role_path }}/templates/nginx-proxy.json" - dest: "{{ grafana_config_path }}/dashboards/" - mode: 0644 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - when: grafana_enabled - - name: Ensure matrix-prometheus-nginxlog-exporter.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2" dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-nginxlog-exporter.service" mode: 0644 - register: matrix_prometheus_nginxlog_exporter_systemd_service_result diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/templates/grafana/nginx-proxy.json b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/grafana/nginx-proxy.json new file mode 100644 index 00000000..395314fd --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/grafana/nginx-proxy.json @@ -0,0 +1,1705 @@ +{ + "__inputs": [ + { + "name": "DS_PROMETHEUS", + "label": "Prometheus", + "description": "", + "type": "datasource", + "pluginId": "prometheus", + "pluginName": "Prometheus" + } + ], + "__elements": {}, + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "9.3.1" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + }, + { + "type": "panel", + "id": "timeseries", + "name": "Time series", + "version": "" + } + ], + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "target": { + "limit": 100, + "matchAny": false, + "tags": [], + "type": "dashboard" + }, + "type": "dashboard" + } + ] + }, + "description": "", + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": 5, + "links": [], + "liveNow": false, + "panels": [ + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 23, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 1 + }, + "id": 20, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for) (rate(http_response_count_total{status=~\"1.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "1xx by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 1 + }, + "id": 21, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"1.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "1xx by uri [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 9 + }, + "id": 18, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for) (rate(http_response_count_total{status=~\"2.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "2xx by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 9 + }, + "id": 19, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"2.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "2xx by uri [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 17 + }, + "id": 16, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for) (rate(http_response_count_total{status=~\"3.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "3xx by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 17 + }, + "id": 17, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"3.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "3xx by uri [$request_host]", + "type": "timeseries" + } + ], + "title": "1xx - 2xx - 3xx Status Code .....................................................", + "type": "row" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 1 + }, + "id": 2, + "panels": [], + "title": "4xx Status Code ......................................................................", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 2 + }, + "id": 9, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for) (rate(http_response_count_total{status=~\"404\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "404 by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 2 + }, + "id": 6, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"404\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "404 by uri [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 10 + }, + "id": 14, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for) (rate(http_response_count_total{status=~\"4.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "4xx by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 10 + }, + "id": 15, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"4.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "4xx by uri [$request_host]", + "type": "timeseries" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 18 + }, + "id": 8, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 3 + }, + "id": 10, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for, status) (rate(http_response_count_total{status=~\"504|502\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "502-504 by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 3 + }, + "id": 11, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_host, status) (rate(http_response_count_total{status=~\"504|502\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "502-504 by host [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 11 + }, + "id": 12, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for) (rate(http_response_count_total{status=~\"5.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "5xx by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 11 + }, + "id": 13, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_host) (rate(http_response_count_total{status=~\"5.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "5xx by uri [$request_host]", + "type": "timeseries" + } + ], + "title": "5xx Status Code ......................................................................", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 19 + }, + "id": 27, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 4 + }, + "id": 24, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(http_x_forwarded_for) (rate(http_response_count_total{status=~\"[6-9]{1}.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "6xx > by upstream addr[$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 4 + }, + "id": 25, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"[6-9]{1}.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "6xx > by uri [$request_host]", + "type": "timeseries" + } + ], + "title": "6xx > Status Code ...................................................................", + "type": "row" + } + ], + "refresh": "30s", + "schemaVersion": 37, + "style": "dark", + "tags": [], + "templating": { + "list": [ + { + "current": { + "selected": true, + "text": "Prometheus", + "value": "Prometheus" + }, + "hide": 0, + "includeAll": false, + "multi": false, + "name": "DS_PROMETHEUS", + "options": [], + "query": "prometheus", + "queryValue": "", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "type": "datasource" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(http_response_count_total, request_host)", + "hide": 0, + "includeAll": true, + "multi": true, + "name": "request_host", + "options": [], + "query": { + "query": "label_values(http_response_count_total, request_host)", + "refId": "StandardVariableQuery" + }, + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "sort": 1, + "type": "query" + } + ] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "NGINX PROXY", + "uid": "x2_jWNF4k", + "version": 12, + "weekStart": "" +} \ No newline at end of file From 4fe601546413d87e7c571f82d084e60648d103c3 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Wed, 15 Feb 2023 09:02:36 +0000 Subject: [PATCH 167/382] Update ntfy 1.30.1 -> 1.31.0 --- roles/custom/matrix-ntfy/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-ntfy/defaults/main.yml b/roles/custom/matrix-ntfy/defaults/main.yml index 397efb35..b2d2f25f 100644 --- a/roles/custom/matrix-ntfy/defaults/main.yml +++ b/roles/custom/matrix-ntfy/defaults/main.yml @@ -7,7 +7,7 @@ matrix_ntfy_base_path: "{{ matrix_base_data_path }}/ntfy" matrix_ntfy_config_dir_path: "{{ matrix_ntfy_base_path }}/config" matrix_ntfy_data_path: "{{ matrix_ntfy_base_path }}/data" -matrix_ntfy_version: v1.30.1 +matrix_ntfy_version: v1.31.0 matrix_ntfy_docker_image: "{{ matrix_container_global_registry_prefix }}binwiederhier/ntfy:{{ matrix_ntfy_version }}" matrix_ntfy_docker_image_force_pull: "{{ matrix_ntfy_docker_image.endswith(':latest') }}" From bb7895678cdaa4eace98654c2558ce99c5f34a32 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Feb 2023 11:47:57 +0200 Subject: [PATCH 168/382] Fix typo --- group_vars/matrix_servers | 2 +- .../tasks/setup_uninstall.yml | 7 +------ 2 files changed, 2 insertions(+), 7 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 78e15eca..a7e229d4 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2480,7 +2480,7 @@ matrix_nginx_proxy_systemd_wanted_services_list: | + (['matrix-client-hydrogen.service'] if matrix_client_hydrogen_enabled else []) + - ([(grafana_idenifier + '.service')] if grafana_enabled else []) + ([(grafana_identifier + '.service')] if grafana_enabled else []) + (['matrix-dimension.service'] if matrix_dimension_enabled else []) + diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml index 6fce44ae..7d27cbae 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml @@ -28,12 +28,7 @@ path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-nginxlog-exporter.service" state: absent - - name: Ensure matrix-prometheus-nginxlog-exporter paths don't exist + - name: Ensure matrix-prometheus-nginxlog-exporter path doesn't exist ansible.builtin.file: path: "{{ matrix_prometheus_nginxlog_exporter_base_path }}" state: absent - - - name: Ensure matrix-prometheus-nginxlog-exporter grafana dashboard doesn't exist - ansible.builtin.file: - path: "{{ grafana_config_path }}/dashboards/nginx-proxy.json" - state: absent From 8a471b97a14bf17d3fec0ee2bae7ba016543c208 Mon Sep 17 00:00:00 2001 From: Stuart Mumford Date: Wed, 15 Feb 2023 17:14:09 +0000 Subject: [PATCH 169/382] Update configuring-playbook-bridge-mautrix-slack.md --- docs/configuring-playbook-bridge-mautrix-slack.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/configuring-playbook-bridge-mautrix-slack.md b/docs/configuring-playbook-bridge-mautrix-slack.md index d0a60aa7..f53fbd10 100644 --- a/docs/configuring-playbook-bridge-mautrix-slack.md +++ b/docs/configuring-playbook-bridge-mautrix-slack.md @@ -8,8 +8,6 @@ The playbook can install and configure [mautrix-slack](https://github.com/mautri See the project's [documentation](https://docs.mau.fi/bridges/go/slack/index.html) to learn what it does and why it might be useful to you. -Note that as of Oct 2022, support for multiple Matrix users using the bot is incomplete. Different users do not yet share the bridged channels. Everyone gets their own copy. - See the [features and roadmap](https://github.com/mautrix/slack/blob/main/ROADMAP.md) for more information. From 93bb117a29a36e4356aac04debb3e6d82875da11 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Feb 2023 08:24:27 +0200 Subject: [PATCH 170/382] Mention Traefik changes causing backward-compatibility break for own-webserver (non-nginx-proxy) users This also updates `docs/configuring-playbook-own-webserver.md`, trying to make it up-to-date with all recent changes. --- CHANGELOG.md | 9 +- docs/configuring-playbook-own-webserver.md | 270 +++++++++++---------- 2 files changed, 142 insertions(+), 137 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b47433d7..f6b76e10 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,12 +22,13 @@ You need to **update you roles** (`just roles` or `make roles`) regardless of wh # 2023-02-12 -## Reverse-proxy configuration changes and initial Traefik support +## (Backward Compatibility) Reverse-proxy configuration changes and initial Traefik support **TLDR**: - there's a new `matrix_playbook_reverse_proxy_type` variable (see [roles/custom/matrix-base/defaults/main.yml](roles/custom/matrix-base/defaults/main.yml)), which lets you tell the playbook what reverse-proxy setup you'd like to have. This makes it easier for people who want to do reverse-proxying in other ways. -- the default reverse-proxy (`matrix_playbook_reverse_proxy_type`) is still `playbook-managed-nginx` (via `matrix-nginx-proxy`), for now. **Existing users should not observe any changes** and can stay on this for now. +- the default reverse-proxy (`matrix_playbook_reverse_proxy_type`) is still `playbook-managed-nginx` (via `matrix-nginx-proxy`), for now. **Existing `matrix-nginx-proxy` users should not observe any changes** and can stay on this for now. +- **Users who use their [own other webserver](docs/configuring-playbook-own-webserver.md) (e.g. Apache, etc.) need to change** `matrix_playbook_reverse_proxy_type` to something like `other-on-same-host`, `other-on-another-host` or `other-nginx-non-container` - we now have **optional [Traefik](https://traefik.io/) support**, so you could easily host Matrix and other Traefik-native services in containers on the same server. Traefik support is still experimental (albeit, good enough) and will improve over time. It does work, but certain esoteric features may not be there yet. - **Traefik will become the default reverse-proxy in the near future**. `matrix-nginx-proxy` will either remain as an option, or be completely removed to simplify the playbook @@ -81,7 +82,9 @@ Traefik does not lock important functionality we'd like to use into [plus packag `matrix_playbook_reverse_proxy_type` still defaults to a value of `playbook-managed-nginx`. -Unless we have some regression, **existing users should be able to update their Matrix server and not observe any changes**. Their setup should still remain on nginx and everything should still work as expected. +Unless we have some regression, **existing `matrix-nginx-proxy` users should be able to update their Matrix server and not observe any changes**. Their setup should still remain on nginx and everything should still work as expected. + +**Users using [their own webservers](docs/configuring-playbook-own-webserver.md) will need to change `matrix_playbook_reverse_proxy_type`** to something like `other-on-same-host`, `other-on-another-host` or `other-nginx-non-container`. Previously, they could toggle `matrix_nginx_proxy_enabled` to `false`, and that made the playbook automatically expose services locally. Currently, we only do this if you change the reverse-proxy type to `other-on-same-host`, `other-on-another-host` or `other-nginx-non-container`. #### How do I explicitly switch to Traefik right now? diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 77fa018d..41b1fd83 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -1,145 +1,21 @@ # Using your own webserver, instead of this playbook's nginx proxy (optional, advanced) -**Note**: the playbook is [in the process of moving to Traefik](../CHANGELOG.md#reverse-proxy-configuration-changes-and-initial-traefik-support). The documentation below should be correct, but things will change soon. +**Note**: the playbook is [in the process of moving to Traefik](../CHANGELOG.md#reverse-proxy-configuration-changes-and-initial-traefik-support). The **documentation below may be incomplete or misleading**. By default, this playbook installs its own nginx webserver (called `matrix-nginx-proxy`, in a Docker container) which listens on ports 80 and 443. If that's alright, you can skip this. -If you don't want this playbook's nginx webserver to take over your server's 80/443 ports like that, -and you'd like to use your own webserver (be it nginx, Apache, Varnish Cache, etc.), you can. +Soon, this default will change and the playbook will install its own [Traefik](https://traefik.io/) reverse-proxy instead. -There are **2 ways you can go about it**, if you'd like to use your own webserver: +## Traefik -- [Method 1: Disabling the integrated nginx reverse-proxy webserver](#method-1-disabling-the-integrated-nginx-reverse-proxy-webserver) +[Traefik](https://traefik.io/) will be the default reverse-proxy for the playbook in the near future. -- [Method 2: Fronting the integrated nginx reverse-proxy webserver with another reverse-proxy](#method-2-fronting-the-integrated-nginx-reverse-proxy-webserver-with-another-reverse-proxy) +There are 2 ways to use Traefik with this playbook, as described below. +### Traefik managed by the playbook -## Method 1: Disabling the integrated nginx reverse-proxy webserver - -This method is about completely disabling the integrated nginx reverse-proxy webserver and replicating its behavior using another webserver. - -If that other webserver is `nginx`, you'd be able to include configuration files generated by the playbook into your `nginx` webserver. - -If you'd like to use another webserver (not `nginx`), you'd need to do things manually. We have examples for other webservers below. - -For an alternative (which keeps `matrix-nginx-proxy` around and connects your other reverse-proxy with it), make sure to check Method #2. - -### Preparation - -No matter which external webserver you decide to go with, you'll need to: - -1) Make sure your web server user (something like `http`, `apache`, `www-data`, `nginx`) is part of the `matrix` group. You should run something like this: `usermod -a -G matrix nginx`. This allows your webserver user to access files owned by the `matrix` group. When using an external nginx webserver, this allows it to read configuration files from `/matrix/nginx-proxy/conf.d`. When using another server, it would make other files, such as `/matrix/static-files/.well-known`, accessible to it. - -2) Edit your configuration file (`inventory/host_vars/matrix./vars.yml`) - - to disable the integrated nginx server: - - ```yaml - matrix_nginx_proxy_enabled: false - ``` - - if using an external server on another host, add the `_http_host_bind_port` or `_http_bind_port` variables for the services that will be exposed by the external server on the other host. The actual name of the variable is listed in the `roles//defaults/vars.yml` file for each service. Most variables follow the `_http_host_bind_port` format. - - These variables will make Docker expose the ports on all network interfaces instead of localhost only. - [Keep in mind that there are some security concerns if you simply proxy everything.](https://github.com/matrix-org/synapse/blob/master/docs/reverse_proxy.md#synapse-administration-endpoints) - - Here are the variables required for the default configuration (Synapse and Element) - ``` - matrix_synapse_reverse_proxy_companion_container_client_api_host_bind_port: '0.0.0.0:8008' - matrix_synapse_reverse_proxy_companion_container_federation_api_host_bind_port: '0.0.0.0:8048' - matrix_client_element_container_http_host_bind_port: "0.0.0.0:8765" - ``` - -3) **If you'll manage SSL certificates by yourself**, edit your configuration file (`inventory/host_vars/matrix./vars.yml`) to disable SSL certificate retrieval: - -```yaml -matrix_ssl_retrieval_method: none -``` - -**Note**: During [installation](installing.md), unless you've disabled SSL certificate management (`matrix_ssl_retrieval_method: none`), the playbook would need 80 to be available, in order to retrieve SSL certificates. **Please manually stop your other webserver while installing**. You can start it back up afterwards. - -### Using your own external nginx webserver - -Once you've followed the [Preparation](#preparation) guide above, it's time to set up your external nginx server. - -Even with `matrix_nginx_proxy_enabled: false`, the playbook still generates some helpful files for you in `/matrix/nginx-proxy/conf.d`. -Those configuration files are adapted for use with an external web server (one not running in the container network). - -You can most likely directly use the config files installed by this playbook at: `/matrix/nginx-proxy/conf.d`. Just include them in your own `nginx.conf` like this: `include /matrix/nginx-proxy/conf.d/*.conf;` - -Note that if your nginx version is old, it might not like our default choice of SSL protocols (particularly the fact that the brand new `TLSv1.3` protocol is enabled). You can override the protocol list by redefining the `matrix_nginx_proxy_ssl_protocols` variable. Example: - -```yaml -# Custom protocol list (removing `TLSv1.3`) to suit your nginx version. -matrix_nginx_proxy_ssl_protocols: "TLSv1.2" -``` - -If you are experiencing issues, try updating to a newer version of Nginx. As a data point in May 2021 a user reported that Nginx 1.14.2 was not working for them. They were getting errors about socket leaks. Updating to Nginx 1.19 fixed their issue. - -### Using your own external Apache webserver - -Once you've followed the [Preparation](#preparation) guide above, you can take a look at the [examples/apache](../examples/apache) directory for a sample configuration. - -### Using your own external caddy webserver - -After following the [Preparation](#preparation) guide above, you can take a look at the [examples/caddy](../examples/caddy) directory and [examples/caddy2](../examples/caddy2) directory for a sample configuration for Caddy v1 and v2, respectively. - -### Using your own HAproxy reverse proxy -After following the [Preparation](#preparation) guide above, you can take a look at the [examples/haproxy](../examples/haproxy) directory for a sample configuration. In this case HAproxy is used as a reverse proxy and a simple Nginx container is used to serve statically `.well-known` files. - -### Using another external webserver - -Feel free to look at the [examples/apache](../examples/apache) directory, or the [template files in the matrix-nginx-proxy role](../roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/). - - -## Method 2: Fronting the integrated nginx reverse-proxy webserver with another reverse-proxy - -This method is about leaving the integrated nginx reverse-proxy webserver be, but making it not get in the way (using up important ports, trying to retrieve SSL certificates, etc.). - -If you wish to use another webserver, the integrated nginx reverse-proxy webserver usually gets in the way because it attempts to fetch SSL certificates and binds to ports 80, 443 and 8448 (if Matrix Federation is enabled). - -You can disable such behavior and make the integrated nginx reverse-proxy webserver only serve traffic locally (or over a local network). - -You would need some configuration like this: - -```yaml -matrix_playbook_reverse_proxy_type: other-on-same-host - -# Do not listen for HTTP on port 80 globally (default), listen on the loopback interface. -# If you'd like, you can make it use the local network as well and reverse-proxy from another local machine. -matrix_nginx_proxy_container_http_host_bind_port: '127.0.0.1:81' - -# Likewise, expose the Matrix Federation port on the loopback interface. -# Since `matrix_nginx_proxy_https_enabled` is set to `false`, this federation port will serve HTTP traffic. -# If you'd like, you can make it use the local network as well and reverse-proxy from another local machine. -# -# You'd most likely need to expose it publicly on port 8448 (8449 was chosen for the local port to prevent overlap). -matrix_nginx_proxy_container_federation_host_bind_port: '127.0.0.1:8449' - -# Coturn relies on SSL certificates that have already been obtained. -# Since we don't obtain any certificates in this `other-on-same-host` mode, it won't work by default. -# An alternative is to tweak some of: `matrix_coturn_tls_enabled`, `matrix_coturn_tls_cert_path` and `matrix_coturn_tls_key_path`. -matrix_coturn_enabled: false -``` - -With this, nginx would still be in use, but it would not bother with anything SSL related or with taking up public ports. - -All services would be served locally on `127.0.0.1:81` and `127.0.0.1:8449` (as per the example configuration above). - -You can then set up another reverse-proxy server on ports 80/443/8448 for all of the expected domains and make traffic go to these local ports. -The expected domains vary depending on the services you have enabled (`matrix.DOMAIN` for sure; `element.DOMAIN`, `dimension.DOMAIN` and `jitsi.DOMAIN` are optional). - -### Sample configuration for running behind Traefik 2.0 - -To run behind Traefik, you can: - -- use a [Playbook-managed Traefik installation](#playbook-managed-traefik-installation) -- or, use [Your own Traefik server (not managed by the playbook)](#your-own-traefik-server-not-managed-by-the-playbook) - -#### Playbook-managed Traefik installation - -The playbook can install and manage Traefik for you using the [com.devture.ansible.role.traefik](https://github.com/devture/com.devture.ansible.role.traefik) role. - -It's simplest if you go with this method. You will need the following configuration: +To switch to Traefik now, use configuration like this: ```yaml matrix_playbook_reverse_proxy_type: playbook-managed-traefik @@ -147,9 +23,11 @@ matrix_playbook_reverse_proxy_type: playbook-managed-traefik devture_traefik_ssl_email_address: YOUR_EMAIL_ADDRESS ``` -#### Your own Traefik server (not managed by the playbook) +This will install Traefik in the place of `matrix-nginx-proxy`. Traefik will manage SSL certificates for all services seamlessly. + +**Note**: during the transition period, `matrix-nginx-proxy` will still be installed in local-only mode. Do not be alarmed to see `matrix-nginx-proxy` running even when you've chosen Traefik as your reverse-proxy. In the future, we'll be able to run without nginx, but we're not there yet. -If you'd like to run Traefik yourself, you can use configuration like this: +### Traefik managed by you ```yaml matrix_playbook_reverse_proxy_type: other-traefik-container @@ -161,7 +39,9 @@ In this mode all roles will still have Traefik labels attached. You will, howeve By default, the playbook congiures services use a `web-secure` (443) and `matrix-federation` (8448) entrypoints, as well as a `default` certificate resolver. -Below is some configuration for running Traefik yourself (although we recommend using a [playbook-managed Traefik installation](#playbook-managed-traefik-installation)). +You need to configure 3 entrypoints for your Traefik server: `web` (TCP port `80`), `web-secure` (TCP port `443`) and `matrix-federation` (TCP port `8448`). + +Below is some configuration for running Traefik yourself, although we recommend using [Traefik managed by the playbook](#traefik-managed-by-the-playbook). Note that this configuration on its own does **not** redirect traffic on port 80 (plain HTTP) to port 443 for HTTPS, which may cause some issues, since the built-in Nginx proxy usually does this. If you are not already doing this in Traefik, it can be added to Traefik in a [file provider](https://docs.traefik.io/v2.0/providers/file/) as follows: @@ -217,3 +97,125 @@ networks: traefik: external: true ``` + +## Another webserver + +If you don't wish to use Traefik or `matrix-nginx-proxy`, you can also use your own webserver. + +Doing this is possible, but requires manual work. + +There are 2 ways to go about it: + +- (recommended) [Fronting the integrated reverse-proxy webserver with another reverse-proxy](#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy) - using a playbook-managed reverse-proxy (either `matrix-nginx-proxy` or Traefik), disabling SSL termination for it, exposing this reverse-proxy on a few local ports (e.g. `127.0.0.1:81`, etc.) and forwarding traffic from your own webserver to those few ports + +- (difficult) [Using no reverse-proxy on the Matrix side at all](#using-no-reverse-proxy-on-the-matrix-side-at-all) disabling all playbook-managed reverse-proxies (no `matrix-nginx-proxy`, no Traefik) + + +### Fronting the integrated reverse-proxy webserver with another reverse-proxy + +This method is about leaving the integrated reverse-proxy webserver be, but making it not get in the way (using up important ports, trying to retrieve SSL certificates, etc.). + +If you wish to use another webserver, the integrated nginx reverse-proxy webserver usually gets in the way because it attempts to fetch SSL certificates and binds to ports 80, 443 and 8448 (if Matrix Federation is enabled). + +You can disable such behavior and make the integrated reverse-proxy webserver only serve traffic locally (or over a local network). + +This is the recommended way for using another reverse-proxy, because the integrated one would act as a black box and wire all Matrix services correctly. You would only need to reverse-proxy a few individual domains and ports over to it. + +**For `matrix-nginx-proxy`** fronted by another reverse-proxy, you would need some configuration like this: + +```yaml +# playbook-managed-proxy is the default right now, so we can keep this commented out. +# matrix_playbook_reverse_proxy_type: playbook-managed-nginx + +# Disable SSL certificate retrieval +matrix_ssl_retrieval_method: none + +# Given that we won't be obtaining SSL certificates here, disable HTTPS +matrix_nginx_proxy_https_enabled: false + +# Do not listen for HTTP on port 80 globally (default), listen on the loopback interface. +# If you'd like, you can make it use the local network as well and reverse-proxy from another local machine. +matrix_nginx_proxy_container_http_host_bind_port: '127.0.0.1:81' + +# Likewise, expose the Matrix Federation port on the loopback interface. +# Since `matrix_nginx_proxy_https_enabled` is set to `false`, this federation port will serve HTTP traffic. +# If you'd like, you can make it use the local network as well and reverse-proxy from another local machine. +# +# You'd most likely need to expose it publicly on port 8448 (8449 was chosen for the local port to prevent overlap). +matrix_nginx_proxy_container_federation_host_bind_port: '127.0.0.1:8449' +``` + +**For Traefik** fronted by another reverse-proxy, you would need some configuration like this: + +```yaml +# playbook-managed-proxy is the default right now, so we can keep this commented out. +matrix_playbook_reverse_proxy_type: playbook-managed-traefik + +# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval +devture_traefik_config_entrypoint_web_secure_enabled: false + +devture_traefik_container_web_host_bind_port: '127.0.0.1:81' + +devture_traefik_additional_entrypoints_auto: + - name: matrix-federation + port: "{{ matrix_federation_public_port }}" + host_bind_port: "127.0.0.1:{{ matrix_federation_public_port }}" + config: {} +``` + +If you'll be fronting with a reverse-proxy that lives on another machine (not on the same one as Matrix), you need to replace `127.0.0.1` in the above configurations with `0.0.0.0` or another network interface. + +### Using no reverse-proxy on the Matrix side at all + +Instead of [Fronting the integrated reverse-proxy webserver with another reverse-proxy](#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy), you can also go another way -- completely disabling the playbook-managed reverse-proxy. You would then need to reverse-proxy from your own webserver directly to Matrix services. + +This is more difficult, as you would need to handle the configuration for each service manually. Enabling additional services would come with extra manual work you need to do. + +If your webserver is on the same machine, sure your web server user (something like `http`, `apache`, `www-data`, `nginx`) is part of the `matrix` group. You should run something like this: `usermod -a -G matrix nginx`. This allows your webserver user to access files owned by the `matrix` group. When using an external nginx webserver, this allows it to read configuration files from `/matrix/nginx-proxy/conf.d`. When using another server, it would make other files, such as `/matrix/static-files/.well-known`, accessible to it. + +#### Using your own nginx reverse-proxy running on the same machine + +If you'll be using `nginx` running on the same machine (not in a container), you can make the playbook help you generate configuration for `nginx` with this configuration: + +```yaml +matrix_playbook_reverse_proxy_type: other-nginx-non-container + +# If you will manage SSL certificates yourself, uncomment the line below +# matrix_ssl_retrieval_method: none + +# If you're using an old nginx version, consider using a custom protocol list +# (removing `TLSv1.3` that is enabled by default) to suit your nginx version. +# matrix_nginx_proxy_ssl_protocols: "TLSv1.2" +``` + +You can most likely directly use the config files installed by this playbook at: `/matrix/nginx-proxy/conf.d`. Just include them in your own `nginx.conf` like this: `include /matrix/nginx-proxy/conf.d/*.conf;` + +#### Using your own reverse-proxy running on the same machine or elsewhere + +To reverse-proxy manually for each service, use configuration like this: + +```yaml +# If your reverse-proxy runs on the same machine: +matrix_playbook_reverse_proxy_type: other-on-same-host + +# Or, if it runs on another machine: +# matrix_playbook_reverse_proxy_type: other-on-another-host + +# Or, optionally customize the network interface prefix (note the trailing `:` character). +# For other-on-same-host, the interface defaults to `127.0.0.1:`. +# For other-on-another-host, the interface defaults to `0.0.0.0:`. +# matrix_playbook_service_host_bind_interface_prefix: '192.168.30.4:' +``` + +With this configuration, each service will be exposed on a custom port. Example: + +- Synapse will be exposed on port `8008` +- [Grafana](configuring-playbook-prometheus-grafana.md) will be exposed on port `3000` +- [synapse-admin](configuring-playbook-synapse-admin.md) will be exposed on port `8766` + +You can capture traffic for these services and forward it to their port. +Some of these services are configured with certain default expecations with regard to hostname, path, etc., so it's not completely arbitrary where you can host them (unless you change the defaults). + +For each new playbook service that you enable, you'll need special handling. + +The [`examples/`](../examples/) directory contains examples for various servers: Caddy, Apache, HAproxy, etc. From e80b98c3ad9d4cd2c93f3e906ab2e38ee8eee158 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Feb 2023 09:22:29 +0200 Subject: [PATCH 171/382] Do not mount SSL certificates into Coturn if TLS is disabled for it --- group_vars/matrix_servers | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index a7e229d4..e55bade9 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2051,7 +2051,7 @@ matrix_coturn_container_additional_volumes: | 'dst': '/privkey.pem', 'options': 'ro', }, - ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] else [] + ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] and matrix_coturn_tls_enabled else [] ) + ( @@ -2066,7 +2066,7 @@ matrix_coturn_container_additional_volumes: | 'dst': '/privatekey.key', 'options': 'ro', }, - ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else [] + ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled and matrix_coturn_tls_enabled else [] ) }} @@ -2074,7 +2074,7 @@ matrix_coturn_systemd_required_services_list: | {{ ['docker.service'] + - ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) + ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled and matrix_coturn_tls_enabled else []) }} ###################################################################### From 649f6512ca3850cd48bb301eae7fe7c5a2cfa588 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Thu, 16 Feb 2023 18:02:25 +0000 Subject: [PATCH 172/382] Update mautrix-discord 0.1.0 -> 0.1.1 --- roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml index 0211f8f3..036b4060 100644 --- a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_discord_container_image_self_build: false matrix_mautrix_discord_container_image_self_build_repo: "https://mau.dev/mautrix/discord.git" matrix_mautrix_discord_container_image_self_build_branch: "{{ 'main' if matrix_mautrix_discord_version == 'latest' else matrix_mautrix_discord_version }}" -matrix_mautrix_discord_version: v0.1.0 +matrix_mautrix_discord_version: v0.1.1 # See: https://mau.dev/mautrix/discord/container_registry matrix_mautrix_discord_docker_image: "{{ matrix_mautrix_discord_docker_image_name_prefix }}mautrix/discord:{{ matrix_mautrix_discord_version }}" matrix_mautrix_discord_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_discord_container_image_self_build else 'dock.mau.dev/' }}" From 64ec1db077f87ae6439485b810b5587cc02f1163 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Thu, 16 Feb 2023 18:03:18 +0000 Subject: [PATCH 173/382] Update mautrix-whatsapp 0.8.1 -> 0.8.2 --- roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml index dd628162..04c24295 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_whatsapp_container_image_self_build: false matrix_mautrix_whatsapp_container_image_self_build_repo: "https://mau.dev/mautrix/whatsapp.git" matrix_mautrix_whatsapp_container_image_self_build_branch: "{{ 'master' if matrix_mautrix_whatsapp_version == 'latest' else matrix_mautrix_whatsapp_version }}" -matrix_mautrix_whatsapp_version: v0.8.1 +matrix_mautrix_whatsapp_version: v0.8.2 # See: https://mau.dev/mautrix/whatsapp/container_registry matrix_mautrix_whatsapp_docker_image: "{{ matrix_mautrix_whatsapp_docker_image_name_prefix }}mautrix/whatsapp:{{ matrix_mautrix_whatsapp_version }}" matrix_mautrix_whatsapp_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_whatsapp_container_image_self_build else 'dock.mau.dev/' }}" From 954920dd4fd8bf928e0d082f981a8f90451f5bb3 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Thu, 16 Feb 2023 20:00:00 +0000 Subject: [PATCH 174/382] Update ntfy 1.31.0 -> 2.0.0 --- roles/custom/matrix-ntfy/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-ntfy/defaults/main.yml b/roles/custom/matrix-ntfy/defaults/main.yml index b2d2f25f..e0c90f5f 100644 --- a/roles/custom/matrix-ntfy/defaults/main.yml +++ b/roles/custom/matrix-ntfy/defaults/main.yml @@ -7,7 +7,7 @@ matrix_ntfy_base_path: "{{ matrix_base_data_path }}/ntfy" matrix_ntfy_config_dir_path: "{{ matrix_ntfy_base_path }}/config" matrix_ntfy_data_path: "{{ matrix_ntfy_base_path }}/data" -matrix_ntfy_version: v1.31.0 +matrix_ntfy_version: v2.0.0 matrix_ntfy_docker_image: "{{ matrix_container_global_registry_prefix }}binwiederhier/ntfy:{{ matrix_ntfy_version }}" matrix_ntfy_docker_image_force_pull: "{{ matrix_ntfy_docker_image.endswith(':latest') }}" From 38c4e464c1a415d526829b2aa56947abcea449b0 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 09:20:22 +0200 Subject: [PATCH 175/382] Fix self-check for Hydrogen and Cinny when running under a subpath --- roles/custom/matrix-client-cinny/tasks/self_check.yml | 6 +++--- roles/custom/matrix-client-hydrogen/tasks/self_check.yml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/roles/custom/matrix-client-cinny/tasks/self_check.yml b/roles/custom/matrix-client-cinny/tasks/self_check.yml index c1c6195e..2ea64a32 100644 --- a/roles/custom/matrix-client-cinny/tasks/self_check.yml +++ b/roles/custom/matrix-client-cinny/tasks/self_check.yml @@ -1,7 +1,7 @@ --- - ansible.builtin.set_fact: - matrix_client_cinny_url_endpoint_public: "https://{{ matrix_server_fqn_cinny }}/config.json" + matrix_client_cinny_url_endpoint_public: "https://{{ matrix_client_cinny_hostname }}{{ matrix_client_cinny_path_prefix }}/config.json" - name: Check Cinny ansible.builtin.uri: @@ -16,9 +16,9 @@ - name: Fail if Cinny not working ansible.builtin.fail: - msg: "Failed checking Cinny is up at `{{ matrix_server_fqn_cinny }}` (checked endpoint: `{{ matrix_client_cinny_url_endpoint_public }}`). Is Cinny running? Is port 443 open in your firewall? Full error: {{ matrix_client_cinny_self_check_result }}" + msg: "Failed checking Cinny is up at `{{ matrix_client_cinny_hostname }}` (checked endpoint: `{{ matrix_client_cinny_url_endpoint_public }}`). Is Cinny running? Is port 443 open in your firewall? Full error: {{ matrix_client_cinny_self_check_result }}" when: "matrix_client_cinny_self_check_result.failed or 'json' not in matrix_client_cinny_self_check_result" - name: Report working Cinny ansible.builtin.debug: - msg: "Cinny at `{{ matrix_server_fqn_cinny }}` is working (checked endpoint: `{{ matrix_client_cinny_url_endpoint_public }}`)" + msg: "Cinny at `{{ matrix_client_cinny_hostname }}` is working (checked endpoint: `{{ matrix_client_cinny_url_endpoint_public }}`)" diff --git a/roles/custom/matrix-client-hydrogen/tasks/self_check.yml b/roles/custom/matrix-client-hydrogen/tasks/self_check.yml index 478f0dc8..42c1f09b 100644 --- a/roles/custom/matrix-client-hydrogen/tasks/self_check.yml +++ b/roles/custom/matrix-client-hydrogen/tasks/self_check.yml @@ -1,7 +1,7 @@ --- - ansible.builtin.set_fact: - matrix_client_hydrogen_url_endpoint_public: "https://{{ matrix_server_fqn_hydrogen }}/config.json" + matrix_client_hydrogen_url_endpoint_public: "https://{{ matrix_client_hydrogen_hostname }}{{ matrix_client_hydrogen_path_prefix }}/config.json" - name: Check Hydrogen ansible.builtin.uri: @@ -16,9 +16,9 @@ - name: Fail if Hydrogen not working ansible.builtin.fail: - msg: "Failed checking Hydrogen is up at `{{ matrix_server_fqn_hydrogen }}` (checked endpoint: `{{ matrix_client_hydrogen_url_endpoint_public }}`). Is Hydrogen running? Is port 443 open in your firewall? Full error: {{ matrix_client_hydrogen_self_check_result }}" + msg: "Failed checking Hydrogen is up at `{{ matrix_client_hydrogen_hostname }}` (checked endpoint: `{{ matrix_client_hydrogen_url_endpoint_public }}`). Is Hydrogen running? Is port 443 open in your firewall? Full error: {{ matrix_client_hydrogen_self_check_result }}" when: "matrix_client_hydrogen_self_check_result.failed or 'json' not in matrix_client_hydrogen_self_check_result" - name: Report working Hydrogen ansible.builtin.debug: - msg: "Hydrogen at `{{ matrix_server_fqn_hydrogen }}` is working (checked endpoint: `{{ matrix_client_hydrogen_url_endpoint_public }}`)" + msg: "Hydrogen at `{{ matrix_client_hydrogen_hostname }}` is working (checked endpoint: `{{ matrix_client_hydrogen_url_endpoint_public }}`)" From 964aa0e84d82161adf4f3b1d5290d94952be9a85 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 09:54:33 +0200 Subject: [PATCH 176/382] Switch to using an external Ntfy role The newly extracted role also has native Traefik support, so we no longer need to rely on `matrix-nginx-proxy` for reverse-proxying to Ntfy. The new role uses port `80` inside the container (not `8080`, like before), because that's the default assumption of the officially published container image. Using a custom port (like `8080`), means the default healthcheck command (which hardcodes port `80`) doesn't work. Instead of fiddling to override the healthcheck command, we've decided to stick to the default port instead. This only affects the inside-the-container port, not any external ports. The new role also supports adding the network ranges of the container's multiple additional networks as "exempt hosts". Previously, only one network's address range was added to "exempt hosts". --- CHANGELOG.md | 11 ++++ docs/configuring-playbook-ntfy.md | 17 +++--- group_vars/matrix_servers | 45 +++++++++++---- playbooks/matrix.yml | 2 +- requirements.yml | 3 + .../matrix-nginx-proxy/defaults/main.yml | 5 -- .../matrix-nginx-proxy/templates/labels.j2 | 12 ---- .../nginx/conf.d/matrix-ntfy.conf.j2 | 2 +- roles/custom/matrix-ntfy/defaults/main.yml | 56 ------------------- roles/custom/matrix-ntfy/tasks/main.yml | 23 -------- roles/custom/matrix-ntfy/tasks/self_check.yml | 27 --------- .../matrix-ntfy/tasks/setup_install.yml | 38 ------------- .../matrix-ntfy/tasks/setup_uninstall.yml | 25 --------- .../matrix-ntfy/templates/ntfy/server.yml.j2 | 11 ---- .../templates/systemd/matrix-ntfy.service.j2 | 39 ------------- .../defaults/main.yml | 3 + .../tasks/validate_config.yml | 17 +++++- 17 files changed, 78 insertions(+), 258 deletions(-) delete mode 100644 roles/custom/matrix-ntfy/defaults/main.yml delete mode 100644 roles/custom/matrix-ntfy/tasks/main.yml delete mode 100644 roles/custom/matrix-ntfy/tasks/self_check.yml delete mode 100644 roles/custom/matrix-ntfy/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-ntfy/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-ntfy/templates/ntfy/server.yml.j2 delete mode 100644 roles/custom/matrix-ntfy/templates/systemd/matrix-ntfy.service.j2 diff --git a/CHANGELOG.md b/CHANGELOG.md index f6b76e10..493315b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,14 @@ +# 2023-02-17 + +## The matrix-ntfy role lives independently now + +**TLDR**: the `matrix-ntfy` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +The `matrix-ntfy` role (which configures [Ntfy](https://ntfy.sh/)) has been extracted from the playbook and now lives in its [own repository](https://gitlab.com/etke.cc/roles/ntfy). This makes it possible to easily use it in other Ansible playbooks. + +You need to **update you roles** (`just roles` or `make roles`) regardless of whether you're enabling Ntfy or not. If you're making use of Ntfy via this playbook, you will need to update variable references in your `vars.yml` file (`matrix_ntfy_` -> `ntfy_`). + + # 2023-02-15 ## The matrix-grafana role lives independently now diff --git a/docs/configuring-playbook-ntfy.md b/docs/configuring-playbook-ntfy.md index a5aec60b..27d2fb12 100644 --- a/docs/configuring-playbook-ntfy.md +++ b/docs/configuring-playbook-ntfy.md @@ -15,17 +15,20 @@ Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars. ```yaml # Enabling it is the only required setting -matrix_ntfy_enabled: true +ntfy_enabled: true -# Some other options -matrix_server_fqn_ntfy: "ntfy.{{ matrix_domain }}" -matrix_ntfy_configuration_extension_yaml: | - log_level: DEBUG +# This is the default hostname. +# Uncomment the line below and change it, if you'd like. +# matrix_server_fqn_ntfy: "ntfy.{{ matrix_domain }}" + +# Uncomment and change to inject additional configuration options. +# ntfy_configuration_extension_yaml: | +# log_level: DEBUG ``` -For a more complete list of variables that you could override, see `roles/custom/matrix-ntfy/defaults/main.yml`. +For a more complete list of variables that you could override, see the [`defaults/main.yml` file](https://gitlab.com/etke.cc/roles/ntfy/-/blob/main/defaults/main.yml) of the ntfy Ansible role. -For a complete list of ntfy config options that you could put in `matrix_ntfy_configuration_extension_yaml`, see the [ntfy config documentation](https://ntfy.sh/docs/config/#config-options). +For a complete list of ntfy config options that you could put in `ntfy_configuration_extension_yaml`, see the [ntfy config documentation](https://ntfy.sh/docs/config/#config-options). ## Installing diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index e55bade9..32b99629 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -326,7 +326,7 @@ devture_systemd_service_manager_services_list_auto: | + (matrix_ssl_renewal_systemd_units_list | selectattr('applicable') | selectattr('enableable') | list ) + - ([{'name': 'matrix-ntfy.service', 'priority': 800, 'groups': ['matrix', 'ntfy']}] if matrix_ntfy_enabled else []) + ([{'name': (ntfy_identifier + '.service'), 'priority': 800, 'groups': ['matrix', 'ntfy']}] if ntfy_enabled else []) + ([{'name': (devture_postgres_identifier + '.service'), 'priority': 500, 'groups': ['matrix', 'postgres']}] if devture_postgres_enabled else []) + @@ -2379,8 +2379,8 @@ matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain: "{{ matrix_s matrix_nginx_proxy_proxy_matrix_enabled: true matrix_nginx_proxy_proxy_element_enabled: "{{ matrix_client_element_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" -matrix_nginx_proxy_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled }}" -matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled }}" +matrix_nginx_proxy_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" +matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" matrix_nginx_proxy_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" @@ -2389,9 +2389,9 @@ matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" matrix_nginx_proxy_proxy_jitsi_manage_wellknown: "{{ matrix_jitsi_require_well_known }}" -matrix_nginx_proxy_proxy_grafana_enabled: "{{ grafana_enabled }}" +matrix_nginx_proxy_proxy_grafana_enabled: "{{ grafana_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" -matrix_nginx_proxy_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" +matrix_nginx_proxy_proxy_ntfy_enabled: "{{ ntfy_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" @@ -2404,7 +2404,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_e matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" -matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_enabled: "{{ matrix_ntfy_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: "{{ matrix_corporal_enabled and matrix_corporal_http_api_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corporal:41081" @@ -2486,7 +2485,7 @@ matrix_nginx_proxy_systemd_wanted_services_list: | + (['matrix-sygnal.service'] if matrix_sygnal_enabled else []) + - (['matrix-ntfy.service'] if matrix_ntfy_enabled else []) + ([(ntfy_identifier + '.service')] if ntfy_enabled else []) + (['matrix-jitsi.service'] if matrix_jitsi_enabled else []) + @@ -2523,7 +2522,7 @@ matrix_ssl_domains_to_obtain_certificates_for: | + ([matrix_server_fqn_sygnal] if matrix_sygnal_enabled else []) + - ([matrix_server_fqn_ntfy] if matrix_ntfy_enabled else []) + ([ntfy_hostname] if ntfy_enabled else []) + (matrix_bot_postmoogle_domains if matrix_bot_postmoogle_enabled else []) + @@ -2886,17 +2885,39 @@ matrix_sygnal_container_http_host_bind_port: "{{ (matrix_playbook_service_host_b ###################################################################### # -# matrix-ntfy +# etke/ntfy # ###################################################################### -matrix_ntfy_enabled: false +ntfy_enabled: false -matrix_ntfy_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '2586') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +ntfy_identifier: matrix-ntfy + +ntfy_base_path: "{{ matrix_base_data_path }}/ntfy" + +ntfy_uid: "{{ matrix_user_uid }}" +ntfy_gid: "{{ matrix_user_gid }}" + +ntfy_hostname: "{{ matrix_server_fqn_ntfy }}" + +ntfy_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else ntfy_identifier }}" + +ntfy_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + +ntfy_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '2586') if matrix_playbook_service_host_bind_interface_prefix else '' }}" + +ntfy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +ntfy_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +ntfy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" + +ntfy_visitor_request_limit_exempt_hosts_hostnames_auto: | + {{ + [matrix_server_fqn_matrix] + }} ###################################################################### # -# /matrix-ntfy +# /etky/ntfy # ###################################################################### diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index f5e21493..f6a83ccb 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -105,7 +105,7 @@ - custom/matrix-etherpad - custom/matrix-email2matrix - custom/matrix-sygnal - - custom/matrix-ntfy + - galaxy/ntfy - custom/matrix-nginx-proxy - custom/matrix-coturn - custom/matrix-aux diff --git a/requirements.yml b/requirements.yml index 716b7c6e..54c172ec 100644 --- a/requirements.yml +++ b/requirements.yml @@ -42,6 +42,9 @@ - src: git+https://gitlab.com/etke.cc/roles/grafana.git version: v9.3.6-1 +- src: git+https://gitlab.com/etke.cc/roles/ntfy.git + version: v2.0.0-0 + - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: fb09fd26f877372417d5586f1e79e83f983f0bd6 diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 38afb3c4..b656f074 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -91,11 +91,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_hostname: "{{ matrix_se matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_hostname }}`)" -matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_hostname: "{{ matrix_server_fqn_ntfy }}" -matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_hostname }}`)" - # matrix_nginx_proxy_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. # See `../templates/labels.j2` for details. # diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 4565da19..1e60ba6d 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -109,18 +109,6 @@ traefik.http.routers.matrix-nginx-proxy-sygnal.entrypoints={{ matrix_nginx_proxy {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_enabled %} -# ntfy -traefik.http.routers.matrix-nginx-proxy-ntfy.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_rule }} -traefik.http.routers.matrix-nginx-proxy-ntfy.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-ntfy.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_ntfy_tls %} -traefik.http.routers.matrix-nginx-proxy-ntfy.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-ntfy.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - traefik.http.services.matrix-nginx-proxy-web.loadbalancer.server.port=8080 {% if matrix_nginx_proxy_proxy_matrix_federation_api_enabled %} diff --git a/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-ntfy.conf.j2 b/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-ntfy.conf.j2 index fbae47e1..7d5c8a0e 100644 --- a/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-ntfy.conf.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-ntfy.conf.j2 @@ -21,7 +21,7 @@ {% if matrix_nginx_proxy_enabled %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s; - set $backend "matrix-ntfy:8080"; + set $backend "matrix-ntfy:80"; proxy_pass http://$backend; {% else %} {# Generic configuration for use outside of our container setup #} diff --git a/roles/custom/matrix-ntfy/defaults/main.yml b/roles/custom/matrix-ntfy/defaults/main.yml deleted file mode 100644 index e0c90f5f..00000000 --- a/roles/custom/matrix-ntfy/defaults/main.yml +++ /dev/null @@ -1,56 +0,0 @@ ---- -# Project source code URL: https://github.com/binwiederhier/ntfy - -matrix_ntfy_enabled: true - -matrix_ntfy_base_path: "{{ matrix_base_data_path }}/ntfy" -matrix_ntfy_config_dir_path: "{{ matrix_ntfy_base_path }}/config" -matrix_ntfy_data_path: "{{ matrix_ntfy_base_path }}/data" - -matrix_ntfy_version: v2.0.0 -matrix_ntfy_docker_image: "{{ matrix_container_global_registry_prefix }}binwiederhier/ntfy:{{ matrix_ntfy_version }}" -matrix_ntfy_docker_image_force_pull: "{{ matrix_ntfy_docker_image.endswith(':latest') }}" - -# Public facing base URL of the ntfy service -matrix_ntfy_base_url: "https://{{ matrix_server_fqn_ntfy }}" - -# Rate limits - -matrix_ntfy_global_topic_limit: 15000 # default -matrix_ntfy_visitor_subscription_limit: 30 # default -matrix_ntfy_visitor_request_limit_burst: 60 # default -matrix_ntfy_visitor_request_limit_replenish: "5s" # default - - -# Controls whether the container exposes its HTTP port (tcp/80 in the container). -# -# Takes an ":" or "" value (e.g. "127.0.0.1:2586"), or empty string to not expose. -matrix_ntfy_container_http_host_bind_port: '' - -# A list of extra arguments to pass to the container (`docker run` command) -matrix_ntfy_container_extra_arguments: [] - -# Controls whether the self-check feature should validate SSL certificates. -matrix_ntfy_self_check_validate_certificates: true - -# Default ntfy configuration template which covers the generic use case. -# You can customize it by controlling the various variables inside it. -# -# For a more advanced customization, you can extend the default (see `matrix_ntfy_configuration_extension_yaml`) -# or completely replace this variable with your own template. -matrix_ntfy_configuration_yaml: "{{ lookup('template', 'templates/ntfy/server.yml.j2') }}" - -matrix_ntfy_configuration_extension_yaml: | - # Your custom YAML configuration for ntfy goes here. - # This configuration extends the default starting configuration (`matrix_ntfy_configuration_yaml`). - # - # You can override individual variables from the default configuration, or introduce new ones. - # - # If you need something more special, you can take full control by - # completely redefining `matrix_ntfy_configuration_yaml`. - -matrix_ntfy_configuration_extension: "{{ matrix_ntfy_configuration_extension_yaml | from_yaml if matrix_ntfy_configuration_extension_yaml | from_yaml is mapping else {} }}" - -# Holds the final ntfy configuration (a combination of the default and its extension). -# You most likely don't need to touch this variable. Instead, see `matrix_ntfy_configuration_yaml`. -matrix_ntfy_configuration: "{{ matrix_ntfy_configuration_yaml | from_yaml | combine(matrix_ntfy_configuration_extension, recursive=True) }}" diff --git a/roles/custom/matrix-ntfy/tasks/main.yml b/roles/custom/matrix-ntfy/tasks/main.yml deleted file mode 100644 index 85d80841..00000000 --- a/roles/custom/matrix-ntfy/tasks/main.yml +++ /dev/null @@ -1,23 +0,0 @@ ---- - -- block: - - when: matrix_ntfy_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-ntfy - - install-all - - install-ntfy - -- block: - - when: not matrix_ntfy_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-ntfy - -- block: - - when: matrix_ntfy_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check.yml" - tags: - - self-check diff --git a/roles/custom/matrix-ntfy/tasks/self_check.yml b/roles/custom/matrix-ntfy/tasks/self_check.yml deleted file mode 100644 index bdbd666a..00000000 --- a/roles/custom/matrix-ntfy/tasks/self_check.yml +++ /dev/null @@ -1,27 +0,0 @@ ---- - -# Query an arbitrary ntfy topic using ntfy's UnifiedPush topic name syntax. -# Expect an empty response (because we query 'since=1s'). - -- ansible.builtin.set_fact: - matrix_ntfy_url_endpoint_public: "{{ matrix_ntfy_base_url }}/upSELFCHECK123/json?poll=1&since=1s" - -- name: Check ntfy - ansible.builtin.uri: - url: "{{ matrix_ntfy_url_endpoint_public }}" - follow_redirects: none - validate_certs: "{{ matrix_ntfy_self_check_validate_certificates }}" - register: matrix_ntfy_self_check_result - check_mode: false - ignore_errors: true - delegate_to: 127.0.0.1 - become: false - -- name: Fail if ntfy not working - ansible.builtin.fail: - msg: "Failed checking ntfy is up at `{{ matrix_server_fqn_ntfy }}` (checked endpoint: `{{ matrix_ntfy_url_endpoint_public }}`). Is ntfy running? Is port 443 open in your firewall? Full error: {{ matrix_ntfy_self_check_result }}" - when: "matrix_ntfy_self_check_result.failed" - -- name: Report working ntfy - ansible.builtin.debug: - msg: "ntfy at `{{ matrix_server_fqn_ntfy }}` is working (checked endpoint: `{{ matrix_ntfy_url_endpoint_public }}`)" diff --git a/roles/custom/matrix-ntfy/tasks/setup_install.yml b/roles/custom/matrix-ntfy/tasks/setup_install.yml deleted file mode 100644 index 38adb122..00000000 --- a/roles/custom/matrix-ntfy/tasks/setup_install.yml +++ /dev/null @@ -1,38 +0,0 @@ ---- - -- name: Ensure matrix-ntfy image is pulled - community.docker.docker_image: - name: "{{ matrix_ntfy_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_ntfy_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_ntfy_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure matrix-ntfy paths exists - ansible.builtin.file: - path: "{{ item }}" - state: directory - mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - with_items: - - "{{ matrix_ntfy_base_path }}" - - "{{ matrix_ntfy_config_dir_path }}" - - "{{ matrix_ntfy_data_path }}" - -- name: Ensure matrix-ntfy config installed - ansible.builtin.copy: - content: "{{ matrix_ntfy_configuration | to_nice_yaml(indent=2, width=999999) }}" - dest: "{{ matrix_ntfy_config_dir_path }}/server.yml" - mode: 0644 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - -- name: Ensure matrix-ntfy.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-ntfy.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-ntfy.service" - mode: 0644 diff --git a/roles/custom/matrix-ntfy/tasks/setup_uninstall.yml b/roles/custom/matrix-ntfy/tasks/setup_uninstall.yml deleted file mode 100644 index afd826f8..00000000 --- a/roles/custom/matrix-ntfy/tasks/setup_uninstall.yml +++ /dev/null @@ -1,25 +0,0 @@ ---- - -- name: Check existence of matrix-ntfy service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-ntfy.service" - register: matrix_ntfy_service_stat - -- when: matrix_ntfy_service_stat.stat.exists | bool - block: - - name: Ensure matrix-ntfy is stopped - ansible.builtin.service: - name: matrix-ntfy - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-ntfy.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-ntfy.service" - state: absent - - - name: Ensure matrix-ntfy path doesn't exist - ansible.builtin.file: - path: "{{ matrix_ntfy_base_path }}" - state: absent diff --git a/roles/custom/matrix-ntfy/templates/ntfy/server.yml.j2 b/roles/custom/matrix-ntfy/templates/ntfy/server.yml.j2 deleted file mode 100644 index 9815fd6b..00000000 --- a/roles/custom/matrix-ntfy/templates/ntfy/server.yml.j2 +++ /dev/null @@ -1,11 +0,0 @@ -base_url: {{ matrix_ntfy_base_url }} -behind_proxy: true -cache_file: /data/cache.db -listen-http: :8080 - -# Rate Limits -global-topic-limit: {{ matrix_ntfy_global_topic_limit | to_json }} -visitor-subscription-limit: {{ matrix_ntfy_visitor_subscription_limit | to_json }} - -visitor-request-limit-burst: {{ matrix_ntfy_visitor_request_limit_burst | to_json }} -visitor-request-limit-replenish: "{{ matrix_ntfy_visitor_request_limit_replenish }}" diff --git a/roles/custom/matrix-ntfy/templates/systemd/matrix-ntfy.service.j2 b/roles/custom/matrix-ntfy/templates/systemd/matrix-ntfy.service.j2 deleted file mode 100644 index 5c2feac0..00000000 --- a/roles/custom/matrix-ntfy/templates/systemd/matrix-ntfy.service.j2 +++ /dev/null @@ -1,39 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=matrix-ntfy -After=docker.service -Requires=docker.service -DefaultDependencies=no - -[Service] -Type=simple -Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-ntfy 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-ntfy 2>/dev/null || true' - -ExecStart={{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-ntfy \ - --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --cap-drop=ALL \ - --read-only \ - --env NTFY_VISITOR_REQUEST_LIMIT_EXEMPT_HOSTS={{matrix_server_fqn_matrix}},localhost,$(docker network inspect {{matrix_docker_network}} -f "{% raw %}{{ (index .IPAM.Config 0).Subnet }}{% endraw %}") \ - {% for arg in matrix_ntfy_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - --network={{ matrix_docker_network }} \ - {% if matrix_ntfy_container_http_host_bind_port %} - -p {{ matrix_ntfy_container_http_host_bind_port }}:8080 \ - {% endif %} - --mount type=bind,src={{ matrix_ntfy_config_dir_path }},dst=/etc/ntfy,ro \ - --mount type=bind,src={{ matrix_ntfy_data_path }},dst=/data \ - {{ matrix_ntfy_docker_image }} \ - serve' - -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-ntfy 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-ntfy 2>/dev/null || true' -Restart=always -RestartSec=30 -SyslogIdentifier=matrix-ntfy - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml index 03867679..f8ccc88a 100644 --- a/roles/custom/matrix_playbook_migration/defaults/main.yml +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -11,3 +11,6 @@ matrix_playbook_migration_matrix_backup_borg_migration_validation_enabled: true # Controls if (`matrix_grafana` -> `grafana`) validation will run. matrix_playbook_migration_matrix_grafana_migration_validation_enabled: true + +# Controls if (`matrix_ntfy` -> `ntfy`) validation will run. +matrix_playbook_migration_matrix_ntfy_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 2ac04ece..80f10197 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -82,7 +82,7 @@ when: "matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars | length > 0" - when: matrix_playbook_migration_matrix_backup_borg_migration_validation_enabled | bool - block: + block:ntfy - ansible.builtin.set_fact: matrix_playbook_migration_backup_borg_migration_vars: |- {{ vars | dict2items | selectattr('key', 'match', 'matrix_backup_borg_.*') | list | items2dict }} @@ -110,3 +110,18 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_grafana_` -> `grafana_`). We found usage of the following variables: {{ matrix_playbook_migration_grafana_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_grafana_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_ntfy_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_ntfy_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_ntfy_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_ntfy variables + ansible.builtin.fail: + msg: >- + The matrix-ntfy role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/ntfy. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_ntfy_` -> `ntfy_`). + We found usage of the following variables: {{ matrix_playbook_migration_ntfy_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_ntfy_migration_vars | length > 0" From b291459bf33d0a6a43b305dd5add202d417a3c6d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 10:13:34 +0200 Subject: [PATCH 177/382] Fix syntax error --- .../custom/matrix_playbook_migration/tasks/validate_config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 80f10197..c17bec69 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -82,7 +82,7 @@ when: "matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars | length > 0" - when: matrix_playbook_migration_matrix_backup_borg_migration_validation_enabled | bool - block:ntfy + block: - ansible.builtin.set_fact: matrix_playbook_migration_backup_borg_migration_vars: |- {{ vars | dict2items | selectattr('key', 'match', 'matrix_backup_borg_.*') | list | items2dict }} From 0ce2121d01d79be4b6c04a4670b99b7ca2a92b76 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 14:59:22 +0200 Subject: [PATCH 178/382] Fix variable typo in validation task --- roles/custom/matrix-client-element/tasks/validate_config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-element/tasks/validate_config.yml b/roles/custom/matrix-client-element/tasks/validate_config.yml index 658542fb..fdf42df7 100644 --- a/roles/custom/matrix-client-element/tasks/validate_config.yml +++ b/roles/custom/matrix-client-element/tasks/validate_config.yml @@ -35,7 +35,7 @@ You need to define a required configuration setting (`{{ item }}`). when: "vars[item] == ''" with_items: - - matrix_synapse_admin_container_labels_traefik_hostname + - matrix_client_element_container_labels_traefik_hostname - matrix_client_element_container_labels_traefik_path_prefix # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). From 519b32543c5feebc9980dd058d0185b907b458b1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 16:13:38 +0200 Subject: [PATCH 179/382] Add matrix_synapse_container_network and matrix_synapse_container_additional_networks --- roles/custom/matrix-synapse/defaults/main.yml | 10 +++++++++- .../matrix-synapse/tasks/synapse/setup_install.yml | 5 +++++ .../synapse/systemd/matrix-synapse-worker.service.j2 | 9 ++++++++- .../synapse/systemd/matrix-synapse.service.j2 | 11 ++++++++++- 4 files changed, 32 insertions(+), 3 deletions(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 93d1ee02..c86db72e 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -69,6 +69,14 @@ matrix_synapse_container_federation_api_tls_port: 8448 matrix_synapse_container_federation_api_plain_port: 8048 +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_synapse_container_network: "{{ matrix_docker_network }}" + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_synapse_container_additional_networks: [] + # Controls whether the matrix-synapse container exposes the Client/Server API port (tcp/{{ matrix_synapse_container_client_api_port }} in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:8008"), or empty string to not expose. @@ -661,7 +669,7 @@ matrix_synapse_redis_password: "" # # If Redis is available, we prefer to use that, instead of talking over Synapse's custom replication protocol. # -# matrix_synapse_replication_listener_enabled: "{{ matrix_synapse_workers_enabled and not matrix_redis_enabled }}" +# matrix_synapse_replication_listener_enabled: "{{ matrix_synapse_workers_enabled and not redis_enabled }}" # We force-enable this listener for now until we debug why communication via Redis fails. matrix_synapse_replication_listener_enabled: true diff --git a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml index 2c6f05b7..773a7518 100644 --- a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml @@ -122,6 +122,11 @@ dest: "{{ matrix_synapse_config_dir_path }}/{{ matrix_server_fqn_matrix }}.log.config" mode: 0644 +- name: Ensure Synapse container network is created + community.general.docker_network: + name: "{{ matrix_synapse_container_network }}" + driver: bridge + - name: Ensure matrix-synapse.service installed ansible.builtin.template: src: "{{ role_path }}/templates/synapse/systemd/matrix-synapse.service.j2" diff --git a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 index 3d50ac9b..e1e43995 100644 --- a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 @@ -14,7 +14,9 @@ ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_s # Intentional delay, so that the homeserver can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_synapse_worker_container_name }} \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name={{ matrix_synapse_worker_container_name }} \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ @@ -45,6 +47,11 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_synapse_docker_image_final }} \ run -m synapse.app.{{ matrix_synapse_worker_details.app }} -c /data/homeserver.yaml -c /data/{{ matrix_synapse_worker_config_file_name }} +{% for network in matrix_synapse_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} {{ matrix_synapse_worker_container_name }} +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach {{ matrix_synapse_worker_container_name }} ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill {{ matrix_synapse_worker_container_name }} 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_synapse_worker_container_name }} 2>/dev/null || true' diff --git a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 index 7f6c2336..1c23c8ac 100644 --- a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 @@ -30,7 +30,9 @@ ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_s ExecStartPre={{ matrix_host_command_sleep }} 3 {% endif %} -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-synapse \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-synapse \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ @@ -63,8 +65,15 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_synapse_docker_image_final }} \ run -m synapse.app.homeserver -c /data/homeserver.yaml +{% for network in matrix_synapse_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-synapse +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-synapse + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse 2>/dev/null || true' + ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-synapse /bin/sh -c 'kill -HUP 1' Restart=always RestartSec=30 From 990a6369e13f7f70e5bc544f55ea99727c554aaa Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 16:23:59 +0200 Subject: [PATCH 180/382] Switch to using an external Redis role --- CHANGELOG.md | 8 +++ group_vars/matrix_servers | 30 +++++++++--- playbooks/matrix.yml | 2 +- requirements.yml | 3 ++ roles/custom/matrix-redis/defaults/main.yml | 25 ---------- roles/custom/matrix-redis/tasks/main.yml | 17 ------- .../matrix-redis/tasks/setup_install.yml | 49 ------------------- .../matrix-redis/tasks/setup_uninstall.yml | 31 ------------ .../matrix-redis/templates/redis.conf.j2 | 4 -- .../templates/systemd/matrix-redis.service.j2 | 37 -------------- .../defaults/main.yml | 3 ++ .../tasks/validate_config.yml | 15 ++++++ 12 files changed, 52 insertions(+), 172 deletions(-) delete mode 100644 roles/custom/matrix-redis/defaults/main.yml delete mode 100644 roles/custom/matrix-redis/tasks/main.yml delete mode 100644 roles/custom/matrix-redis/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-redis/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-redis/templates/redis.conf.j2 delete mode 100644 roles/custom/matrix-redis/templates/systemd/matrix-redis.service.j2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 493315b7..5fea8b78 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # 2023-02-17 +## The matrix-redis role lives independently now + +**TLDR**: the `matrix-redis` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +The `matrix-redis` role (which configures [Redis](https://redis.io/)) has been extracted from the playbook and now lives in its [own repository](https://gitlab.com/etke.cc/roles/redis). This makes it possible to easily use it in other Ansible playbooks. + +You need to **update you roles** (`just roles` or `make roles`) regardless of whether you're enabling Ntfy or not. If you're making use of Ntfy via this playbook, you will need to update variable references in your `vars.yml` file (`matrix_redis_` -> `redis_`). + ## The matrix-ntfy role lives independently now **TLDR**: the `matrix-ntfy` role is now included from another repository. Some variables have been renamed. All functionality remains intact. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 32b99629..1cd84c6b 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -340,7 +340,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-prometheus-nginxlog-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if matrix_prometheus_nginxlog_exporter_enabled else []) + - ([{'name': 'matrix-redis', 'priority': 750, 'groups': ['matrix', 'redis']}] if matrix_redis_enabled else []) + ([{'name': (redis_identifier + '.service'), 'priority': 750, 'groups': ['matrix', 'redis']}] if redis_enabled else []) + ([{'name': 'matrix-registration.service', 'priority': 4000, 'groups': ['matrix', 'registration']}] if matrix_registration_enabled else []) + @@ -2923,15 +2923,22 @@ ntfy_visitor_request_limit_exempt_hosts_hostnames_auto: | ###################################################################### # -# matrix-redis +# etke/redis # ###################################################################### -matrix_redis_enabled: "{{ matrix_synapse_workers_enabled }}" +redis_enabled: "{{ matrix_synapse_workers_enabled }}" + +redis_identifier: matrix-redis + +redis_uid: "{{ matrix_user_uid }}" +redis_gid: "{{ matrix_user_gid }}" + +redis_base_path: "{{ matrix_base_data_path }}/redis" ###################################################################### # -# /matrix-redis +# /etle/redis # ###################################################################### @@ -3077,7 +3084,12 @@ matrix_synapse_container_metrics_api_host_bind_port: "{{ (matrix_playbook_servic # # For exposing the Synapse Manhole port (plain HTTP) to the local host. matrix_synapse_container_manhole_api_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9000') if matrix_synapse_metrics_enabled and matrix_playbook_service_host_bind_interface_prefix else '' }}" -# + +matrix_synapse_container_additional_networks: | + {{ + ([redis_container_network] if matrix_synapse_redis_enabled and matrix_synapse_redis_host == redis_identifier else []) + }} + # For exposing the Synapse worker (and metrics) ports to the local host. matrix_synapse_workers_container_host_bind_address: "{{ matrix_playbook_service_host_bind_interface_prefix[0:-1] if (matrix_synapse_workers_enabled and matrix_playbook_service_host_bind_interface_prefix) else '' }}" @@ -3134,6 +3146,8 @@ matrix_synapse_systemd_required_services_list: | + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + + ([redis_identifier ~ '.service'] if matrix_synapse_redis_enabled and matrix_synapse_redis_host == redis_identifier else []) + + (['matrix-goofys.service'] if matrix_s3_media_store_enabled else []) }} @@ -3145,9 +3159,9 @@ matrix_synapse_systemd_wanted_services_list: | }} # Synapse workers (used for parallel load-scaling) need Redis for IPC. -matrix_synapse_redis_enabled: "{{ matrix_redis_enabled }}" -matrix_synapse_redis_host: "{{ 'matrix-redis' if matrix_redis_enabled else '' }}" -matrix_synapse_redis_password: "{{ matrix_redis_connection_password if matrix_redis_enabled else '' }}" +matrix_synapse_redis_enabled: "{{ redis_enabled }}" +matrix_synapse_redis_host: "{{ redis_identifier if redis_enabled else '' }}" +matrix_synapse_redis_password: "{{ redis_connection_password if redis_enabled else '' }}" matrix_synapse_container_extra_arguments_auto: "{{ matrix_homeserver_container_extra_arguments_auto }}" matrix_synapse_app_service_config_files_auto: "{{ matrix_homeserver_app_service_config_files_auto }}" diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index f6a83ccb..6980a6f9 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -43,7 +43,7 @@ - role: galaxy/com.devture.ansible.role.postgres - - custom/matrix-redis + - galaxy/redis - custom/matrix-corporal - custom/matrix-bridge-appservice-discord - custom/matrix-bridge-appservice-slack diff --git a/requirements.yml b/requirements.yml index 54c172ec..cc270a6c 100644 --- a/requirements.yml +++ b/requirements.yml @@ -45,6 +45,9 @@ - src: git+https://gitlab.com/etke.cc/roles/ntfy.git version: v2.0.0-0 +- src: git+https://gitlab.com/etke.cc/roles/redis.git + version: v7.0.7-0 + - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: fb09fd26f877372417d5586f1e79e83f983f0bd6 diff --git a/roles/custom/matrix-redis/defaults/main.yml b/roles/custom/matrix-redis/defaults/main.yml deleted file mode 100644 index 779ec3a3..00000000 --- a/roles/custom/matrix-redis/defaults/main.yml +++ /dev/null @@ -1,25 +0,0 @@ ---- -# Project source code URL: https://github.com/redis/redis - -matrix_redis_enabled: true - -matrix_redis_connection_password: "" - -matrix_redis_base_path: "{{ matrix_base_data_path }}/redis" -matrix_redis_data_path: "{{ matrix_redis_base_path }}/data" - -matrix_redis_version: 7.0.7-alpine -matrix_redis_docker_image_v6: "{{ matrix_container_global_registry_prefix }}redis:{{ matrix_redis_version }}" -matrix_redis_docker_image_latest: "{{ matrix_redis_docker_image_v6 }}" -matrix_redis_docker_image_to_use: '{{ matrix_redis_docker_image_latest }}' - -matrix_redis_docker_image_force_pull: "{{ matrix_redis_docker_image_to_use.endswith(':latest') }}" - -# A list of extra arguments to pass to the container -matrix_redis_container_extra_arguments: [] - -# Controls whether the matrix-redis container exposes a port (tcp/6379 in the container) -# that can be used to access redis from outside the container -# -# Takes an ":" or "" value (e.g. "127.0.0.1:6379"), or empty string to not expose. -matrix_redis_container_redis_bind_port: "" diff --git a/roles/custom/matrix-redis/tasks/main.yml b/roles/custom/matrix-redis/tasks/main.yml deleted file mode 100644 index 29640005..00000000 --- a/roles/custom/matrix-redis/tasks/main.yml +++ /dev/null @@ -1,17 +0,0 @@ ---- - -- block: - - when: matrix_redis_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-redis - - install-all - - install-redis - -- block: - - when: not matrix_redis_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-redis diff --git a/roles/custom/matrix-redis/tasks/setup_install.yml b/roles/custom/matrix-redis/tasks/setup_install.yml deleted file mode 100644 index 1384c042..00000000 --- a/roles/custom/matrix-redis/tasks/setup_install.yml +++ /dev/null @@ -1,49 +0,0 @@ ---- - -- name: Ensure redis Docker image is pulled - community.docker.docker_image: - name: "{{ matrix_redis_docker_image_to_use }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_redis_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_redis_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure redis paths exist - ansible.builtin.file: - path: "{{ item }}" - state: directory - mode: 0700 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_username }}" - with_items: - - "{{ matrix_redis_base_path }}" - - "{{ matrix_redis_data_path }}" - -# We do this as a separate task, because: -# - we'd like to do it for the data path only, not for the base path (which contains root-owned environment variable files we'd like to leave as-is) -# - we need to do it without `mode`, or we risk making certain `.conf` and other files's executable bit to flip to true -- name: Ensure redis data path ownership is correct - ansible.builtin.file: - path: "{{ matrix_redis_data_path }}" - state: directory - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_username }}" - recurse: true - -- name: Ensure redis environment variables file created - ansible.builtin.template: - src: "{{ role_path }}/templates/{{ item }}.j2" - dest: "{{ matrix_redis_base_path }}/{{ item }}" - mode: 0644 - with_items: - - "redis.conf" - -- name: Ensure matrix-redis.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-redis.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-redis.service" - mode: 0644 - register: matrix_redis_systemd_service_result diff --git a/roles/custom/matrix-redis/tasks/setup_uninstall.yml b/roles/custom/matrix-redis/tasks/setup_uninstall.yml deleted file mode 100644 index e2bbc9e4..00000000 --- a/roles/custom/matrix-redis/tasks/setup_uninstall.yml +++ /dev/null @@ -1,31 +0,0 @@ ---- - -- name: Check existence of matrix-redis service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-redis.service" - register: matrix_redis_service_stat - -- when: matrix_redis_service_stat.stat.exists | bool - block: - - name: Ensure matrix-redis is stopped - ansible.builtin.service: - name: matrix-redis - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-redis.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-redis.service" - state: absent - -- name: Check existence of matrix-redis local data path - ansible.builtin.stat: - path: "{{ matrix_redis_data_path }}" - register: matrix_redis_data_path_stat - -# We just want to notify the user. Deleting data is too destructive. -- name: Notify if matrix-redis local data remains - ansible.builtin.debug: - msg: "Note: You are not using a local redis instance, but some old data remains from before in `{{ matrix_redis_data_path }}`. Feel free to delete it." - when: matrix_redis_data_path_stat.stat.exists | bool diff --git a/roles/custom/matrix-redis/templates/redis.conf.j2 b/roles/custom/matrix-redis/templates/redis.conf.j2 deleted file mode 100644 index 34371356..00000000 --- a/roles/custom/matrix-redis/templates/redis.conf.j2 +++ /dev/null @@ -1,4 +0,0 @@ -#jinja2: lstrip_blocks: "True" -{% if matrix_redis_connection_password %} -requirepass {{ matrix_redis_connection_password }} -{% endif %} diff --git a/roles/custom/matrix-redis/templates/systemd/matrix-redis.service.j2 b/roles/custom/matrix-redis/templates/systemd/matrix-redis.service.j2 deleted file mode 100644 index 5f6699f8..00000000 --- a/roles/custom/matrix-redis/templates/systemd/matrix-redis.service.j2 +++ /dev/null @@ -1,37 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=Matrix Redis server -After=docker.service -Requires=docker.service - -[Service] -Type=simple -ExecStartPre=-/usr/bin/docker stop matrix-redis -ExecStartPre=-/usr/bin/docker rm matrix-redis - -ExecStart=/usr/bin/docker run --rm --name matrix-redis \ - --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --cap-drop=ALL \ - --read-only \ - --tmpfs=/tmp:rw,noexec,nosuid,size=100m \ - --network={{ matrix_docker_network }} \ - {% if matrix_redis_container_redis_bind_port %} - -p {{ matrix_redis_container_redis_bind_port }}:6379 \ - {% endif %} - --mount type=bind,src={{ matrix_redis_base_path }}/redis.conf,dst=/usr/local/etc/redis/redis.conf,ro \ - --mount type=bind,src={{ matrix_redis_data_path }},dst=/data \ - {% for arg in matrix_redis_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - {{ matrix_redis_docker_image_to_use }} \ - redis-server /usr/local/etc/redis/redis.conf - -ExecStop=-/usr/bin/docker stop matrix-redis -ExecStop=-/usr/bin/docker rm matrix-redis -Restart=always -RestartSec=30 -SyslogIdentifier=matrix-redis - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml index f8ccc88a..514125ec 100644 --- a/roles/custom/matrix_playbook_migration/defaults/main.yml +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -14,3 +14,6 @@ matrix_playbook_migration_matrix_grafana_migration_validation_enabled: true # Controls if (`matrix_ntfy` -> `ntfy`) validation will run. matrix_playbook_migration_matrix_ntfy_migration_validation_enabled: true + +# Controls if (`matrix_redis` -> `redis`) validation will run. +matrix_playbook_migration_matrix_redis_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index c17bec69..7afd2154 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -125,3 +125,18 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_ntfy_` -> `ntfy_`). We found usage of the following variables: {{ matrix_playbook_migration_ntfy_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_ntfy_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_redis_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_redis_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_redis_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_redis variables + ansible.builtin.fail: + msg: >- + The matrix-redis role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/redis. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_redis_` -> `redis_`). + We found usage of the following variables: {{ matrix_playbook_migration_redis_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_redis_migration_vars | length > 0" From 154d077ec736eb776ca01a3d20b6d7263d9be875 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 16:24:10 +0200 Subject: [PATCH 181/382] Upgrade ntfy (v2.0.0-0 -> v2.0.0-1) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index cc270a6c..e1f419fa 100644 --- a/requirements.yml +++ b/requirements.yml @@ -43,7 +43,7 @@ version: v9.3.6-1 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git - version: v2.0.0-0 + version: v2.0.0-1 - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.7-0 From 632026513e577127ff06bd9cc845cf88ada28192 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 17:16:50 +0200 Subject: [PATCH 182/382] Add matrix_synapse_uid, matrix_synapse_gid and matrix_synapse_username --- group_vars/matrix_servers | 4 ++++ roles/custom/matrix-synapse/defaults/main.yml | 4 ++++ .../ext/encryption-disabler/setup_install.yml | 4 ++-- .../ext/mjolnir-antispam/setup_install.yml | 2 +- .../tasks/ext/rest-auth/setup_install.yml | 4 ++-- .../ext/s3-storage-provider/setup_install.yml | 4 ++-- .../ext/shared-secret-auth/setup_install.yml | 4 ++-- .../synapse-simple-antispam/setup_install.yml | 2 +- .../tasks/goofys/setup_install.yml | 4 ++-- .../tasks/import_media_store.yml | 4 ++-- .../compress_room.yml | 2 +- .../rust-synapse-compress-state/main.yml | 4 ++-- .../matrix-synapse/tasks/setup_install.yml | 4 ++-- .../tasks/synapse/setup_install.yml | 20 +++++++++---------- .../workers/util/setup_files_for_worker.yml | 4 ++-- .../matrix-synapse/tasks/validate_config.yml | 14 ++++++++----- .../goofys/systemd/matrix-goofys.service.j2 | 2 +- .../systemd/matrix-synapse-worker.service.j2 | 2 +- .../synapse/systemd/matrix-synapse.service.j2 | 2 +- 19 files changed, 51 insertions(+), 39 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 1cd84c6b..e47f1e60 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3071,6 +3071,10 @@ matrix_client_cinny_self_check_validate_certificates: "{{ false if matrix_playbo matrix_synapse_enabled: "{{ matrix_homeserver_implementation == 'synapse' }}" +matrix_synapse_username: "{{ matrix_user_username }}" +matrix_synapse_uid: "{{ matrix_user_uid }}" +matrix_synapse_gid: "{{ matrix_user_gid }}" + matrix_synapse_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" # When ma1sd is enabled, we can use it to validate phone numbers. It's something that the homeserver cannot do by itself. diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index c86db72e..86f1f23e 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -4,6 +4,10 @@ matrix_synapse_enabled: true +matrix_synapse_username: '' +matrix_synapse_uid: '' +matrix_synapse_gid: '' + matrix_synapse_container_image_self_build: false matrix_synapse_container_image_self_build_repo: "https://github.com/matrix-org/synapse.git" diff --git a/roles/custom/matrix-synapse/tasks/ext/encryption-disabler/setup_install.yml b/roles/custom/matrix-synapse/tasks/ext/encryption-disabler/setup_install.yml index 6ba5946d..3725545f 100644 --- a/roles/custom/matrix-synapse/tasks/ext/encryption-disabler/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/ext/encryption-disabler/setup_install.yml @@ -6,8 +6,8 @@ dest: "{{ matrix_synapse_ext_path }}/matrix_e2ee_filter.py" force: true mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" register: result retries: "{{ devture_playbook_help_geturl_retries_count }}" delay: "{{ devture_playbook_help_geturl_retries_delay }}" diff --git a/roles/custom/matrix-synapse/tasks/ext/mjolnir-antispam/setup_install.yml b/roles/custom/matrix-synapse/tasks/ext/mjolnir-antispam/setup_install.yml index 5d36a234..f3218808 100644 --- a/roles/custom/matrix-synapse/tasks/ext/mjolnir-antispam/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/ext/mjolnir-antispam/setup_install.yml @@ -11,7 +11,7 @@ version: "{{ matrix_synapse_ext_spam_checker_mjolnir_antispam_git_version }}" dest: "{{ matrix_synapse_ext_path }}/mjolnir" become: true - become_user: "{{ matrix_user_username }}" + become_user: "{{ matrix_synapse_username }}" - ansible.builtin.set_fact: matrix_synapse_spam_checker: > diff --git a/roles/custom/matrix-synapse/tasks/ext/rest-auth/setup_install.yml b/roles/custom/matrix-synapse/tasks/ext/rest-auth/setup_install.yml index 3c4d8cb5..ad58830a 100644 --- a/roles/custom/matrix-synapse/tasks/ext/rest-auth/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/ext/rest-auth/setup_install.yml @@ -11,8 +11,8 @@ dest: "{{ matrix_synapse_ext_path }}/rest_auth_provider.py" force: true mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" register: result retries: "{{ devture_playbook_help_geturl_retries_count }}" delay: "{{ devture_playbook_help_geturl_retries_delay }}" diff --git a/roles/custom/matrix-synapse/tasks/ext/s3-storage-provider/setup_install.yml b/roles/custom/matrix-synapse/tasks/ext/s3-storage-provider/setup_install.yml index c892e0b6..384f1681 100644 --- a/roles/custom/matrix-synapse/tasks/ext/s3-storage-provider/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/ext/s3-storage-provider/setup_install.yml @@ -12,8 +12,8 @@ path: "{{ item }}" state: directory mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" with_items: - "{{ matrix_synapse_ext_s3_storage_provider_base_path }}" - "{{ matrix_synapse_ext_s3_storage_provider_bin_path }}" diff --git a/roles/custom/matrix-synapse/tasks/ext/shared-secret-auth/setup_install.yml b/roles/custom/matrix-synapse/tasks/ext/shared-secret-auth/setup_install.yml index 6dead736..c974bd2c 100644 --- a/roles/custom/matrix-synapse/tasks/ext/shared-secret-auth/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/ext/shared-secret-auth/setup_install.yml @@ -16,8 +16,8 @@ dest: "{{ matrix_synapse_ext_path }}/shared_secret_authenticator.py" force: true mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" register: result retries: "{{ devture_playbook_help_geturl_retries_count }}" delay: "{{ devture_playbook_help_geturl_retries_delay }}" diff --git a/roles/custom/matrix-synapse/tasks/ext/synapse-simple-antispam/setup_install.yml b/roles/custom/matrix-synapse/tasks/ext/synapse-simple-antispam/setup_install.yml index 15fe220a..e62c62df 100644 --- a/roles/custom/matrix-synapse/tasks/ext/synapse-simple-antispam/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/ext/synapse-simple-antispam/setup_install.yml @@ -16,7 +16,7 @@ version: "{{ matrix_synapse_ext_spam_checker_synapse_simple_antispam_git_version }}" dest: "{{ matrix_synapse_ext_path }}/synapse-simple-antispam" become: true - become_user: "{{ matrix_user_username }}" + become_user: "{{ matrix_synapse_username }}" - ansible.builtin.set_fact: matrix_synapse_modules: > diff --git a/roles/custom/matrix-synapse/tasks/goofys/setup_install.yml b/roles/custom/matrix-synapse/tasks/goofys/setup_install.yml index 7fd4659d..dd6d3e92 100644 --- a/roles/custom/matrix-synapse/tasks/goofys/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/goofys/setup_install.yml @@ -27,8 +27,8 @@ path: "{{ matrix_s3_media_store_path }}" state: directory mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" when: "not local_path_matrix_s3_media_store_path_stat.failed and not local_path_matrix_s3_media_store_path_stat.stat.exists" - name: Ensure goofys environment variables file created diff --git a/roles/custom/matrix-synapse/tasks/import_media_store.yml b/roles/custom/matrix-synapse/tasks/import_media_store.yml index bdd99ca4..a6c08587 100644 --- a/roles/custom/matrix-synapse/tasks/import_media_store.yml +++ b/roles/custom/matrix-synapse/tasks/import_media_store.yml @@ -66,8 +66,8 @@ - name: Ensure media store permissions are correct (generic case) ansible.builtin.file: path: "{{ matrix_synapse_media_store_path }}" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" recurse: true when: "not matrix_s3_media_store_enabled | bool" diff --git a/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml b/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml index 4eafad1f..1d71fea6 100644 --- a/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml +++ b/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml @@ -7,7 +7,7 @@ ansible.builtin.set_fact: matrix_synapse_rust_synapse_compress_state_compress_room_command: >- {{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-rust-synapse-compress-state-compress-room - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} --cap-drop=ALL --network={{ matrix_docker_network }} --mount type=bind,src={{ matrix_synapse_rust_synapse_compress_state_base_path }},dst=/work diff --git a/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/main.yml b/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/main.yml index 45716ef0..8f781e51 100644 --- a/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/main.yml +++ b/roles/custom/matrix-synapse/tasks/rust-synapse-compress-state/main.yml @@ -39,8 +39,8 @@ path: "{{ matrix_synapse_rust_synapse_compress_state_base_path }}" state: directory mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" - name: Ensure rust-synapse-compress-state image is pulled community.docker.docker_image: diff --git a/roles/custom/matrix-synapse/tasks/setup_install.yml b/roles/custom/matrix-synapse/tasks/setup_install.yml index 921df753..c8b191ec 100644 --- a/roles/custom/matrix-synapse/tasks/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/setup_install.yml @@ -5,8 +5,8 @@ path: "{{ item.path }}" state: directory mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" with_items: - {path: "{{ matrix_synapse_config_dir_path }}", when: true} - {path: "{{ matrix_synapse_ext_path }}", when: true} diff --git a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml index 773a7518..7a7b5ed3 100644 --- a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml @@ -14,8 +14,8 @@ path: "{{ matrix_synapse_media_store_path }}" state: directory mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" when: "not local_path_media_store_stat.failed and not local_path_media_store_stat.stat.exists" - when: "matrix_synapse_container_image_self_build | bool" @@ -27,7 +27,7 @@ version: "{{ matrix_synapse_docker_image.split(':')[1] }}" force: "yes" become: true - become_user: "{{ matrix_user_username }}" + become_user: "{{ matrix_synapse_username }}" register: matrix_synapse_git_pull_results - name: Check if Synapse Docker image exists @@ -68,8 +68,8 @@ ansible.builtin.template: src: "{{ role_path }}/templates/synapse/customizations/Dockerfile.j2" dest: "{{ matrix_synapse_customized_docker_src_files_path }}/Dockerfile" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" mode: 0640 - name: Ensure customized Docker image for Synapse is built @@ -98,7 +98,7 @@ docker run --rm --name=matrix-config - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} --cap-drop=ALL --mount type=bind,src={{ matrix_synapse_config_dir_path }},dst=/data -e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml @@ -113,8 +113,8 @@ content: "{{ matrix_synapse_configuration | to_nice_yaml(indent=2, width=999999) }}" dest: "{{ matrix_synapse_config_dir_path }}/homeserver.yaml" mode: 0644 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" - name: Ensure Synapse log config installed ansible.builtin.template: @@ -143,7 +143,7 @@ ansible.builtin.template: src: "{{ role_path }}/templates/synapse/prometheus/external_prometheus.yml.example.j2" dest: "{{ matrix_synapse_base_path }}/external_prometheus.yml.example" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" mode: 0644 when: matrix_synapse_metrics_proxying_enabled | bool diff --git a/roles/custom/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml b/roles/custom/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml index 6910445e..d3509412 100644 --- a/roles/custom/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml +++ b/roles/custom/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml @@ -10,8 +10,8 @@ src: "{{ role_path }}/templates/synapse/worker.yaml.j2" dest: "{{ matrix_synapse_config_dir_path }}/{{ matrix_synapse_worker_config_file_name }}" mode: 0644 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" + owner: "{{ matrix_synapse_uid }}" + group: "{{ matrix_synapse_gid }}" - name: Ensure systemd service exists for {{ matrix_synapse_worker_systemd_service_name }} ansible.builtin.template: diff --git a/roles/custom/matrix-synapse/tasks/validate_config.yml b/roles/custom/matrix-synapse/tasks/validate_config.yml index 56f12bf6..31a10e91 100644 --- a/roles/custom/matrix-synapse/tasks/validate_config.yml +++ b/roles/custom/matrix-synapse/tasks/validate_config.yml @@ -6,11 +6,15 @@ You need to define a required configuration setting (`{{ item }}`) for using Synapse. when: "vars[item] == ''" with_items: - - "matrix_synapse_macaroon_secret_key" - - "matrix_synapse_database_host" - - "matrix_synapse_database_user" - - "matrix_synapse_database_password" - - "matrix_synapse_database_database" + - matrix_synapse_username + - matrix_synapse_uid + - matrix_synapse_gid + - matrix_synapse_container_network + - matrix_synapse_macaroon_secret_key + - matrix_synapse_database_host + - matrix_synapse_database_user + - matrix_synapse_database_password + - matrix_synapse_database_database - name: Fail if asking for more than 1 instance of single-instance workers ansible.builtin.fail: diff --git a/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 b/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 index a968965b..fea02d47 100644 --- a/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 +++ b/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 @@ -13,7 +13,7 @@ ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} rm %n ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name %n \ --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} \ --mount type=bind,src=/etc/passwd,dst=/etc/passwd,ro \ --mount type=bind,src=/etc/group,dst=/etc/group,ro \ --mount type=bind,src={{ matrix_s3_media_store_path }},dst=/s3,bind-propagation=shared \ diff --git a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 index e1e43995..de475bb4 100644 --- a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 @@ -18,7 +18,7 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --rm \ --name={{ matrix_synapse_worker_container_name }} \ --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} \ --cap-drop=ALL \ --read-only \ --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_synapse_tmp_directory_size_mb }}m \ diff --git a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 index 1c23c8ac..a287f553 100644 --- a/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 @@ -34,7 +34,7 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --rm \ --name=matrix-synapse \ --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} \ --cap-drop=ALL \ --read-only \ --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_synapse_tmp_directory_size_mb }}m \ From 977521885037d739066aea67858f0118fee195cb Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 18:31:01 +0200 Subject: [PATCH 183/382] Add matrix_synapse_email_app_name variable --- roles/custom/matrix-synapse/defaults/main.yml | 1 + .../custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 86f1f23e..9842f31f 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -706,6 +706,7 @@ matrix_synapse_email_smtp_user: "" matrix_synapse_email_smtp_pass: "" matrix_synapse_email_smtp_require_transport_security: false matrix_synapse_email_notif_from: "Matrix " +matrix_synapse_email_app_name: Matrix matrix_synapse_email_client_base_url: "https://{{ matrix_server_fqn_element }}" matrix_synapse_email_invite_client_location: "https://app.element.io" diff --git a/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 index 1885aeef..1afbcdc3 100644 --- a/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -2363,7 +2363,7 @@ email: # subjects. It defaults to 'Matrix'. # #app_name: my_branded_matrix_server - app_name: Matrix + app_name: {{ matrix_synapse_email_app_name | to_json }} # Uncomment the following to enable sending emails for messages that the user # has missed. Disabled by default. From 5df89a44b386cfba65310742e3e2f4f6bd0cd6fe Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Feb 2023 19:37:34 +0200 Subject: [PATCH 184/382] Add support for customizing Synapse templates --- CHANGELOG.md | 6 +++ docs/configuring-playbook-synapse.md | 39 +++++++++++++++ roles/custom/matrix-synapse/defaults/main.yml | 47 ++++++++++++++++++- .../tasks/synapse/setup_install.yml | 3 +- .../matrix-synapse/tasks/validate_config.yml | 19 ++++++++ .../synapse/customizations/Dockerfile.j2 | 44 +++++++++++++++++ .../templates/synapse/homeserver.yaml.j2 | 3 ++ 7 files changed, 159 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5fea8b78..25d35138 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # 2023-02-17 +## Synapse templates customization support + +The playbook can now help you customize Synapse's templates. + +Additional details are available in the [Customizing templates](docs/configuring-playbook-synapse.md#customizing-templates) section of our Synapse documentation. + ## The matrix-redis role lives independently now **TLDR**: the `matrix-redis` role is now included from another repository. Some variables have been renamed. All functionality remains intact. diff --git a/docs/configuring-playbook-synapse.md b/docs/configuring-playbook-synapse.md index 4dc8f519..ef562ae8 100644 --- a/docs/configuring-playbook-synapse.md +++ b/docs/configuring-playbook-synapse.md @@ -80,3 +80,42 @@ matrix_synapse_configuration_extension_yaml: | backchannel_logout_enabled: true # Optional ``` + +## Customizing templates + +[Templates](https://github.com/matrix-org/synapse/blob/develop/docs/templates.md) are used by Synapse for showing **certain web pages** handled by the server, as well as for **email notifications**. + +This playbook allows you to customize the default templates (see the [`synapse/res/templates` directory](https://github.com/matrix-org/synapse/tree/develop/synapse/res/templates)). + +If template customization is enabled, the playbook will build a custom container image based on the official one. + +Your custom templates need to live in a public or private git repository. This repository will be cloned during Synapse image customization (during the playbook run). + +To enable template customizations, use a configuration (`inventory/host_vars/matrix.DOMAIN/vars.yml`) like this: + +```yaml +# If you'd like to ensure that the customized image is built each time the playbook runs, enable this. +# Otherwise, the customized image will only be rebuilt whenever the Synapse version changes (once every ~2 weeks). +# matrix_synapse_docker_image_customized_build_nocache: true + +matrix_synapse_container_image_customizations_templates_enabled: true + +# Our templates live in a templates/ directory within the repository. +# If they're at the root path, delete this line. +matrix_synapse_container_image_customizations_templates_in_container_template_files_relative_path: templates + +matrix_synapse_container_image_customizations_templates_git_repository_url: git@github.com:organization/repository.git +matrix_synapse_container_image_customizations_templates_git_repository_branch: main + +matrix_synapse_container_image_customizations_templates_git_repository_keyscan_enabled: true +matrix_synapse_container_image_customizations_templates_git_repository_keyscan_hostname: github.com + +# If your git repository is public, do not define the private key (remove the variable). +matrix_synapse_container_image_customizations_templates_git_repository_ssh_private_key: | + -----BEGIN OPENSSH PRIVATE KEY----- + .... + -----END OPENSSH PRIVATE KEY----- +``` + +As mentioned in Synapse's Templates documentation, Synapse will fall back to its own templates if a template is not found in that directory. +Due to this, it's recommended to only store and maintain template files in your repository if you need to make custom changes. Other files (which you don't need to change), should not be duplicated, so that you don't need to worry about getting out-of-sync with the original Synapse templates. diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 9842f31f..e4e2e5e2 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -21,12 +21,41 @@ matrix_synapse_container_image_self_build_repo: "https://github.com/matrix-org/s # - `matrix_synapse_container_image_customizations_dockerfile_body_custom` # - `matrix_synapse_docker_image_customized` # - `matrix_synapse_docker_image_final` -matrix_synapse_container_image_customizations_enabled: "{{ matrix_synapse_ext_synapse_s3_storage_provider_enabled }}" +matrix_synapse_container_image_customizations_enabled: |- + {{ + matrix_synapse_container_image_customizations_s3_storage_provider_installation_enabled + or + matrix_synapse_container_image_customizations_templates_enabled + }} # Controls whether custom build steps will be added to the Dockerfile for installing s3-storage-provider. # The version that will be installed is specified in `matrix_synapse_ext_synapse_s3_storage_provider_version`. matrix_synapse_container_image_customizations_s3_storage_provider_installation_enabled: "{{ matrix_synapse_ext_synapse_s3_storage_provider_enabled }}" +# Controls whether custom build steps will be added to the Dockerfile for customizing the email templates used by Synapse. +# +# Example usage: +# +# ```yml +# matrix_synapse_container_image_customizations_templates_enabled: true +# # The templates are expected to be in a `templates/` subdirectory in +# matrix_synapse_container_image_customizations_templates_in_container_template_files_relative_path: templates/ +# matrix_synapse_container_image_customizations_templates_git_repository_url: git@github.com:organization/repository.git +# matrix_synapse_container_image_customizations_templates_git_repository_branch: main +# matrix_synapse_container_image_customizations_templates_git_repository_keyscan_enabled: true +# matrix_synapse_container_image_customizations_templates_git_repository_keyscan_hostname: github.com +# ``` +# +# See: https://github.com/matrix-org/synapse/blob/develop/docs/templates.md +matrix_synapse_container_image_customizations_templates_enabled: false +matrix_synapse_container_image_customizations_templates_in_container_base_path: /custom-templates +matrix_synapse_container_image_customizations_templates_in_container_template_files_relative_path: '' +matrix_synapse_container_image_customizations_templates_in_container_full_path: "{{ matrix_synapse_container_image_customizations_templates_in_container_base_path }}/{{ matrix_synapse_container_image_customizations_templates_in_container_template_files_relative_path }}" +matrix_synapse_container_image_customizations_templates_git_repository_url: '' +matrix_synapse_container_image_customizations_templates_git_repository_branch: main +matrix_synapse_container_image_customizations_templates_git_repository_keyscan_enabled: false +matrix_synapse_container_image_customizations_templates_git_repository_keyscan_hostname: '' + # matrix_synapse_container_image_customizations_dockerfile_body contains your custom Dockerfile steps # for building your customized Synapse image based on the original (upstream) image (`matrix_synapse_docker_image`). # A `FROM ...` clause is included automatically so you don't have to. @@ -49,6 +78,15 @@ matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith # This image will be based on the upstream `matrix_synapse_docker_image` image, only if `matrix_synapse_container_image_customizations_enabled: true`. matrix_synapse_docker_image_customized: "localhost/matrixdotorg/synapse:{{ matrix_synapse_docker_image_tag }}-customized" +# Controls whether the customized image (`matrix_synapse_docker_image_customized`) is to be force-built without layer caching enabled. +# This is useful if you've enabled customizations (e.g. `matrix_synapse_container_image_customizations_templates_enabled`), +# which clone some branch of some repository, and you'd like for each Ansible run to pull new revisions from that branch. +matrix_synapse_docker_image_customized_build_nocache: false + +# Controls whether the customized image (`matrix_synapse_docker_image_customized`) is to be built, even if it already exists. +# Related to: matrix_synapse_docker_image_customized_build_nocache +matrix_synapse_docker_image_customized_force_source: "{{ matrix_synapse_docker_image_customized_build_nocache }}" + # matrix_synapse_docker_image_final holds the name of the Synapse image to run depending on whether or not customizations are enabled. matrix_synapse_docker_image_final: "{{ matrix_synapse_docker_image_customized if matrix_synapse_container_image_customizations_enabled else matrix_synapse_docker_image }} " @@ -230,6 +268,13 @@ matrix_synapse_rc_federation: matrix_synapse_federation_rr_transactions_per_room_per_second: 50 +# Controls the templates directory setting. +# +# See: +# - `matrix_synapse_container_image_customizations_templates_enabled` +# - https://github.com/matrix-org/synapse/blob/develop/docs/templates.md +matrix_synapse_templates_custom_template_directory: "{{ matrix_synapse_container_image_customizations_templates_in_container_full_path if matrix_synapse_container_image_customizations_templates_enabled else '' }}" + # Controls whether the TLS federation listener is enabled (tcp/8448). # Only makes sense if federation is enabled (`matrix_synapse_federation_enabled`). # Note that federation may potentially be enabled as non-TLS on `matrix_synapse_container_federation_api_plain_port` as well. diff --git a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml index 7a7b5ed3..4a62d88d 100644 --- a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml @@ -76,10 +76,11 @@ community.docker.docker_image: name: "{{ matrix_synapse_docker_image_customized }}" source: build + force_source: "{{ matrix_synapse_docker_image_customized_force_source }}" build: dockerfile: Dockerfile path: "{{ matrix_synapse_customized_docker_src_files_path }}" - pull: true + nocache: "{{ matrix_synapse_docker_image_customized_build_nocache }}" - name: Check if a Synapse signing key exists ansible.builtin.stat: diff --git a/roles/custom/matrix-synapse/tasks/validate_config.yml b/roles/custom/matrix-synapse/tasks/validate_config.yml index 31a10e91..607c75b8 100644 --- a/roles/custom/matrix-synapse/tasks/validate_config.yml +++ b/roles/custom/matrix-synapse/tasks/validate_config.yml @@ -76,3 +76,22 @@ when: "item.old in matrix_synapse_configuration_extension" with_items: - {'old': 'federation_ip_range_blacklist', 'new': 'ip_range_blacklist'} + +- when: matrix_synapse_container_image_customizations_templates_enabled | bool + block: + - name: Fail if required `matrix_synapse_container_image_customizations_templates_*` settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`) when enabling `matrix_synapse_container_image_customizations_templates_enabled`. + when: "vars[item] == ''" + with_items: + - matrix_synapse_container_image_customizations_templates_git_repository_url + - matrix_synapse_container_image_customizations_templates_git_repository_branch + + - name: Fail if required `matrix_synapse_container_image_customizations_templates_git_repository_keyscan_*` settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`) when enabling `matrix_synapse_container_image_customizations_templates_git_repository_keyscan`. + when: "matrix_synapse_container_image_customizations_templates_git_repository_keyscan_enabled | bool and vars[item] == ''" + with_items: + - matrix_synapse_container_image_customizations_templates_git_repository_keyscan_hostname diff --git a/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2 b/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2 index 3919e955..6257f198 100644 --- a/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2 @@ -1,7 +1,51 @@ +#jinja2: lstrip_blocks: "True" FROM {{ matrix_synapse_docker_image }} {% if matrix_synapse_container_image_customizations_s3_storage_provider_installation_enabled %} RUN pip install synapse-s3-storage-provider=={{ matrix_synapse_ext_synapse_s3_storage_provider_version }} {% endif %} +{% if matrix_synapse_container_image_customizations_templates_enabled %} +{# +This ugly script below does quite a lot: + - installs git and other dependencies temporarily, just so we could do a shallow-clone + - prepare the SSH config: keyscanning (if enabled), private key (if enabled) + - performs a git shallow clone with just the branch we need + - makes sure the files are owned by the user that will actually run the container later + - removes the `.git` directory to save space, but keeps git revision in `git-revision.txt`, should we need it for debugging + - finally, verifies that the templates path can indeed be found within the base path (sanity check) +#} +{% set dependencies = ['git', 'ssh', 'openssh-client'] %} +RUN \ + {% if matrix_synapse_container_image_customizations_templates_git_repository_ssh_private_key %} + echo '{{ matrix_synapse_container_image_customizations_templates_git_repository_ssh_private_key | b64encode }}' | base64 -d > /custom-templates-private-key && \ + chmod 400 /custom-templates-private-key && \ + {% endif %} + apt-get update && \ + apt-get install --no-install-recommends -y {{ dependencies | join(' ') }} && \ + {% if matrix_synapse_container_image_customizations_templates_git_repository_keyscan_enabled %} + mkdir ~/.ssh && \ + chmod 700 ~/.ssh && \ + ssh-keyscan -t rsa {{ matrix_synapse_container_image_customizations_templates_git_repository_keyscan_hostname }} >> ~/.ssh/known_hosts && \ + {% endif %} + {% if matrix_synapse_container_image_customizations_templates_git_repository_ssh_private_key %}GIT_SSH_COMMAND='ssh -i /custom-templates-private-key'{% endif %} git \ + clone \ + --branch={{ matrix_synapse_container_image_customizations_templates_git_repository_branch }} \ + --depth=1 \ + --single-branch \ + --no-tags \ + {{ matrix_synapse_container_image_customizations_templates_git_repository_url }} \ + {{ matrix_synapse_container_image_customizations_templates_in_container_base_path }} && \ + /bin/sh -c 'cd {{ matrix_synapse_container_image_customizations_templates_in_container_base_path }} && git rev-parse HEAD > git-revision.txt' && \ + rm -rf {{ matrix_synapse_container_image_customizations_templates_in_container_base_path }}/.git && \ + chown -R {{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} {{ matrix_synapse_container_image_customizations_templates_in_container_base_path }} && \ + apt-get autoremove -y {{ dependencies | join(' ') }} && \ + {% if matrix_synapse_container_image_customizations_templates_git_repository_ssh_private_key %} + rm /custom-templates-private-key && \ + {% endif %} + true + +RUN /bin/sh -c 'stat {{ matrix_synapse_container_image_customizations_templates_in_container_base_path }}/{{ matrix_synapse_container_image_customizations_templates_in_container_template_files_relative_path }} || exit 1' +{% endif %} + {{ matrix_synapse_container_image_customizations_dockerfile_body_custom }} diff --git a/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 index 1afbcdc3..9b039ff5 100644 --- a/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -555,6 +555,9 @@ templates: # information about using custom templates. # #custom_template_directory: /path/to/custom/templates/ + {% if matrix_synapse_templates_custom_template_directory %} + custom_template_directory: {{ matrix_synapse_templates_custom_template_directory | to_json }} + {% endif %} # List of rooms to exclude from sync responses. This is useful for server # administrators wishing to group users into a room without these users being able From 89f2fa53c08166c5eacf7a78b47fa3c98af337bc Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Fri, 17 Feb 2023 15:48:53 -0500 Subject: [PATCH 185/382] remove comment regarding nginx in traefik doc --- docs/configuring-playbook-own-webserver.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 41b1fd83..d1975dae 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -148,7 +148,6 @@ matrix_nginx_proxy_container_federation_host_bind_port: '127.0.0.1:8449' **For Traefik** fronted by another reverse-proxy, you would need some configuration like this: ```yaml -# playbook-managed-proxy is the default right now, so we can keep this commented out. matrix_playbook_reverse_proxy_type: playbook-managed-traefik # Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval From 886d1cddd0aad2dc085230c6a923515f824485cb Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sat, 18 Feb 2023 19:10:23 +0000 Subject: [PATCH 186/382] Update ntfy v2.0.0-1 -> v2.0.1-0 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index e1f419fa..66089d47 100644 --- a/requirements.yml +++ b/requirements.yml @@ -43,7 +43,7 @@ version: v9.3.6-1 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git - version: v2.0.0-1 + version: v2.0.1-0 - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.7-0 From 0da308e24d9e3ad5d6b368e367d72190992d1d39 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 19 Feb 2023 10:19:54 +0200 Subject: [PATCH 187/382] Upgrade com.devture.ansible.role.traefik --- CHANGELOG.md | 2 +- docs/configuring-playbook-own-webserver.md | 2 +- requirements.yml | 2 +- roles/custom/matrix-base/defaults/main.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 25d35138..5dd5e9f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -118,7 +118,7 @@ Unless we have some regression, **existing `matrix-nginx-proxy` users should be ```yaml matrix_playbook_reverse_proxy_type: playbook-managed-traefik -devture_traefik_ssl_email_address: YOUR_EMAIL_ADDRESS +devture_traefik_config_certificatesResolvers_acme_email: YOUR_EMAIL_ADDRESS ``` You may still need to keep certain old `matrix_nginx_proxy_*` variables (like `matrix_nginx_proxy_base_domain_serving_enabled`), even when using Traefik. For now, we recommend keeping all `matrix_nginx_proxy_*` variables just in case. In the future, reliance on `matrix-nginx-proxy` will be removed. diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index d1975dae..6377b8b1 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -20,7 +20,7 @@ To switch to Traefik now, use configuration like this: ```yaml matrix_playbook_reverse_proxy_type: playbook-managed-traefik -devture_traefik_ssl_email_address: YOUR_EMAIL_ADDRESS +devture_traefik_config_certificatesResolvers_acme_email: YOUR_EMAIL_ADDRESS ``` This will install Traefik in the place of `matrix-nginx-proxy`. Traefik will manage SSL certificates for all services seamlessly. diff --git a/requirements.yml b/requirements.yml index 66089d47..ed0faa52 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.7-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: fb09fd26f877372417d5586f1e79e83f983f0bd6 + version: da90b6cdbbb9374ca72c1b530867add2fd8f024e - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index e72142c3..da5459f8 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -283,7 +283,7 @@ matrix_homeserver_app_service_config_files_auto: [] # - `playbook-managed-traefik` # - the playbook will install devture-traefik # - Traefik will do SSL termination, unless you disable it (e.g. `devture_traefik_config_entrypoint_web_secure_enabled: false`) -# - if SSL termination is enabled (as it is by default), you need to populate: `devture_traefik_ssl_email_address` +# - if SSL termination is enabled (as it is by default), you need to populate: `devture_traefik_config_certificatesResolvers_acme_email` # - it will also install matrix-nginx-proxy in local-only mode, while we migrate the rest of the services to a Traefik-native mode of working # # - `playbook-managed-nginx` From a758301bf6658382b20cc5d028140afd4bae5262 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 19 Feb 2023 10:20:40 +0200 Subject: [PATCH 188/382] Upgrade geerlingguy.docker (6.0.4 -> 6.1.0) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index ed0faa52..3f48cec2 100644 --- a/requirements.yml +++ b/requirements.yml @@ -1,7 +1,7 @@ --- - src: geerlingguy.docker - version: 6.0.4 + version: 6.1.0 - src: git+https://github.com/devture/com.devture.ansible.role.docker_sdk_for_python.git version: 6ba3be490b6f4c6f35ea109aeb8e533fa231b3a5 From 5e7f30a129077004baaa701dea1aad571f176e8d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 19 Feb 2023 11:20:58 +0200 Subject: [PATCH 189/382] Fix appservice-discord/appservice-slack/appservice-webhooks port troubles with external reverse-proxy Continuation of 6cda711 --- group_vars/matrix_servers | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index e47f1e60..f148e042 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -455,7 +455,7 @@ matrix_appservice_discord_enabled: false # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-discord over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-discord's client-server port to the local host. -matrix_appservice_discord_container_http_host_bind_port: "{{ matrix_playbook_service_host_bind_interface_prefix + ':9005' if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_appservice_discord_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9005') if matrix_playbook_service_host_bind_interface_prefix else '' }}" # If the homeserver disables presence, it's likely better (less wasteful) to also disable presence on the bridge side. matrix_appservice_discord_bridge_disablePresence: "{{ not matrix_synapse_presence_enabled }}" @@ -502,7 +502,7 @@ matrix_appservice_webhooks_container_image_self_build: "{{ matrix_architecture ! # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-webhooks over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-webhooks' client-server port to the local host. -matrix_appservice_webhooks_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '' ~ matrix_appservice_webhooks_matrix_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_appservice_webhooks_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ matrix_appservice_webhooks_matrix_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_appservice_webhooks_appservice_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'webhook.as.token', rounds=655555) | to_uuid }}" @@ -541,7 +541,7 @@ matrix_appservice_slack_container_image_self_build: "{{ matrix_architecture not # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-slack over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-slack's client-server port to the local host. -matrix_appservice_slack_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix + ':' ~ matrix_appservice_slack_slack_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_appservice_slack_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ matrix_appservice_slack_slack_port) if matrix_playbook_service_host_bind_interface_prefix else '' }}" matrix_appservice_slack_appservice_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'slack.as.token', rounds=655555) | to_uuid }}" From 6438f1ff427cfbb604399c23e25f4371eb671cdd Mon Sep 17 00:00:00 2001 From: Tilo Spannagel Date: Sun, 19 Feb 2023 17:16:17 +0100 Subject: [PATCH 190/382] Add matrix_playbook_ssl_enabled: true to configuring-playbook-own-webserver.md Otherwise the public hookshot url will use http and not https --- docs/configuring-playbook-own-webserver.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 6377b8b1..e7dc2455 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -127,6 +127,9 @@ This is the recommended way for using another reverse-proxy, because the integra # playbook-managed-proxy is the default right now, so we can keep this commented out. # matrix_playbook_reverse_proxy_type: playbook-managed-nginx +# Ensure that public urls use https +matrix_playbook_ssl_enabled: true + # Disable SSL certificate retrieval matrix_ssl_retrieval_method: none @@ -150,6 +153,9 @@ matrix_nginx_proxy_container_federation_host_bind_port: '127.0.0.1:8449' ```yaml matrix_playbook_reverse_proxy_type: playbook-managed-traefik +# Ensure that public urls use https +matrix_playbook_ssl_enabled: true + # Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval devture_traefik_config_entrypoint_web_secure_enabled: false From 6cdbde01d7c2fa980e18853d0118e554e10def86 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Feb 2023 16:50:39 +0200 Subject: [PATCH 191/382] Fix various Traefik headers syntax `always` is an nginx suffix, which shouldn't have been added to these. --- roles/custom/matrix-client-cinny/defaults/main.yml | 4 ++-- roles/custom/matrix-client-element/defaults/main.yml | 4 ++-- roles/custom/matrix-client-hydrogen/defaults/main.yml | 4 ++-- roles/custom/matrix-synapse-admin/defaults/main.yml | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/roles/custom/matrix-client-cinny/defaults/main.yml b/roles/custom/matrix-client-cinny/defaults/main.yml index 5e5f216f..f1114c36 100644 --- a/roles/custom/matrix-client-cinny/defaults/main.yml +++ b/roles/custom/matrix-client-cinny/defaults/main.yml @@ -95,11 +95,11 @@ matrix_client_cinny_http_header_content_security_policy: frame-ancestors 'self' # Specifies the value of the `Permission-Policy` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy -matrix_client_cinny_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_client_cinny_floc_optout_enabled else '' }}" +matrix_client_cinny_http_header_content_permission_policy: "{{ 'interest-cohort=()' if matrix_client_cinny_floc_optout_enabled else '' }}" # Specifies the value of the `Strict-Transport-Security` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security -matrix_client_cinny_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_client_cinny_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" +matrix_client_cinny_http_header_strict_transport_security: "max-age=31536000; includeSubDomains{{ '; preload' if matrix_client_cinny_hsts_preload_enabled else '' }}" # Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses # diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 4ab72bab..c9540ae0 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -98,11 +98,11 @@ matrix_client_element_http_header_content_security_policy: frame-ancestors 'self # Specifies the value of the `Permission-Policy` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy -matrix_client_element_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_client_element_floc_optout_enabled else '' }}" +matrix_client_element_http_header_content_permission_policy: "{{ 'interest-cohort=()' if matrix_client_element_floc_optout_enabled else '' }}" # Specifies the value of the `Strict-Transport-Security` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security -matrix_client_element_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_client_element_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" +matrix_client_element_http_header_strict_transport_security: "max-age=31536000; includeSubDomains{{ '; preload' if matrix_client_element_hsts_preload_enabled else '' }}" # Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses # diff --git a/roles/custom/matrix-client-hydrogen/defaults/main.yml b/roles/custom/matrix-client-hydrogen/defaults/main.yml index 35d18cd9..f3ddb81b 100644 --- a/roles/custom/matrix-client-hydrogen/defaults/main.yml +++ b/roles/custom/matrix-client-hydrogen/defaults/main.yml @@ -94,11 +94,11 @@ matrix_client_hydrogen_http_header_content_security_policy: frame-ancestors 'sel # Specifies the value of the `Permission-Policy` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy -matrix_client_hydrogen_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_client_hydrogen_floc_optout_enabled else '' }}" +matrix_client_hydrogen_http_header_content_permission_policy: "{{ 'interest-cohort=()' if matrix_client_hydrogen_floc_optout_enabled else '' }}" # Specifies the value of the `Strict-Transport-Security` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security -matrix_client_hydrogen_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_client_hydrogen_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" +matrix_client_hydrogen_http_header_strict_transport_security: "max-age=31536000; includeSubDomains{{ '; preload' if matrix_client_hydrogen_hsts_preload_enabled else '' }}" # Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses # diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 91383a9e..ae912f71 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -102,11 +102,11 @@ matrix_synapse_admin_http_header_content_security_policy: frame-ancestors 'self' # Specifies the value of the `Permission-Policy` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy -matrix_synapse_admin_http_header_content_permission_policy: "{{ 'interest-cohort=() always' if matrix_synapse_admin_floc_optout_enabled else '' }}" +matrix_synapse_admin_http_header_content_permission_policy: "{{ 'interest-cohort=()' if matrix_synapse_admin_floc_optout_enabled else '' }}" # Specifies the value of the `Strict-Transport-Security` header. # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security -matrix_synapse_admin_http_header_strict_transport_security: "{{ '\"max-age=31536000; includeSubDomains; preload\" always' if matrix_synapse_admin_hsts_preload_enabled else '\"max-age=31536000; includeSubDomains\" always' }}" +matrix_synapse_admin_http_header_strict_transport_security: "max-age=31536000; includeSubDomains{{ '; preload' if matrix_synapse_admin_hsts_preload_enabled else '' }}" # Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses # From 8e592fb0c1761610e8dd79ba948ac5df18fc3328 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Feb 2023 16:53:19 +0200 Subject: [PATCH 192/382] Upgrade grafana role (v9.3.6-1 -> v9.3.6-2) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 3f48cec2..67a1efc2 100644 --- a/requirements.yml +++ b/requirements.yml @@ -40,7 +40,7 @@ version: v1.2.3-1.7.6-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git - version: v9.3.6-1 + version: v9.3.6-2 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git version: v2.0.1-0 From ba429f6a403131339c940684dc5a1e678558b53c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bergr=C3=BCbe?= <68869895+Bergruebe@users.noreply.github.com> Date: Mon, 20 Feb 2023 16:59:38 +0100 Subject: [PATCH 193/382] Added matrix_nginx_proxy_https_enabled to the own-nginx documentation --- docs/configuring-playbook-own-webserver.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index e7dc2455..34e0bf3e 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -185,6 +185,9 @@ If you'll be using `nginx` running on the same machine (not in a container), you ```yaml matrix_playbook_reverse_proxy_type: other-nginx-non-container +# If you want https configured in /matrix/nginx-proxy/conf.d/ +matrix_nginx_proxy_https_enabled: true + # If you will manage SSL certificates yourself, uncomment the line below # matrix_ssl_retrieval_method: none From 601b67c02d45f9bae711e0e6655c60b597d8a05c Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Mon, 20 Feb 2023 22:21:02 +0100 Subject: [PATCH 194/382] Update Draupnir Config --- .../templates/production.yaml.j2 | 78 ++++++++++--------- 1 file changed, 43 insertions(+), 35 deletions(-) diff --git a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 index dcc61b1b..2dbff886 100644 --- a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 +++ b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 @@ -1,79 +1,79 @@ -# Endpoint URL that draupnir uses to interact with the matrix homeserver (client-server API), +# Endpoint URL that Draupnir uses to interact with the matrix homeserver (client-server API), # set this to the pantalaimon URL if you're using that. homeserverUrl: "{{ matrix_homeserver_url }}" -# Endpoint URL that draupnir could use to fetch events related to reports (client-server API and /_synapse/), +# Endpoint URL that Draupnir could use to fetch events related to reports (client-server API and /_synapse/), # only set this to the public-internet homeserver client API URL, do NOT set this to the pantalaimon URL. rawHomeserverUrl: "{{ matrix_homeserver_url }}" -# Matrix Access Token to use, draupnir will only use this if pantalaimon.use is false. +# Matrix Access Token to use, Draupnir will only use this if pantalaimon.use is false. accessToken: "{{ matrix_bot_draupnir_access_token }}" # Options related to Pantalaimon (https://github.com/matrix-org/pantalaimon) #pantalaimon: -# # Whether or not draupnir will use pantalaimon to access the matrix homeserver, +# # Whether or not Draupnir will use pantalaimon to access the matrix homeserver, # # set to `true` if you're using pantalaimon. # # # # Be sure to point homeserverUrl to the pantalaimon instance. # # -# # draupnir will log in using the given username and password once, +# # Draupnir will log in using the given username and password once, # # then store the resulting access token in a file under dataPath. # use: false # # # The username to login with. # username: draupnir # -# # The password draupnir will login with. +# # The password Draupnir will login with. # # # # After successfully logging in once, this will be ignored, so this value can be blanked after first startup. # password: your_password -# The path draupnir will store its state/data in, leave default ("/data/storage") when using containers. +# The path Draupnir will store its state/data in, leave default ("/data/storage") when using containers. dataPath: "/data" -# If true (the default), draupnir will only accept invites from users present in managementRoom. +# If true (the default), Draupnir will only accept invites from users present in managementRoom. autojoinOnlyIfManager: true # If `autojoinOnlyIfManager` is false, only the members in this space can invite # the bot to new rooms. -#acceptInvitesFromSpace: "!example:example.org" +acceptInvitesFromSpace: "!example:example.org" -# Whether draupnir should report ignored invites to the management room (if autojoinOnlyIfManager is true). +# Whether Draupnir should report ignored invites to the management room (if autojoinOnlyIfManager is true). recordIgnoredInvites: false -# The room ID (or room alias) of the management room, anyone in this room can issue commands to draupnir. +# The room ID (or room alias) of the management room, anyone in this room can issue commands to Draupnir. # -# draupnir has no more granular access controls other than this, be sure you trust everyone in this room - secure it! +# Draupnir has no more granular access controls other than this, be sure you trust everyone in this room - secure it! # # This should be a room alias or room ID - not a matrix.to URL. # -# Note: By default, draupnir is fairly verbose - expect a lot of messages in this room. +# Note: By default, Draupnir is fairly verbose - expect a lot of messages in this room. # (see verboseLogging to adjust this a bit.) -managementRoom: "{{ matrix_bot_draupnir_management_room }}" +managementRoom: "#moderators:example.org" -# Whether draupnir should log a lot more messages in the room, -# mainly involves "all-OK" messages, and debugging messages for when draupnir checks bans in a room. -verboseLogging: false +# Whether Draupnir should log a lot more messages in the room, +# mainly involves "all-OK" messages, and debugging messages for when Draupnir checks bans in a room. +verboseLogging: true # The log level of terminal (or container) output, # can be one of DEBUG, INFO, WARN and ERROR, in increasing order of importance and severity. # -# This should be at INFO or DEBUG in order to get support for draupnir problems. +# This should be at INFO or DEBUG in order to get support for Draupnir problems. logLevel: "INFO" -# Whether or not draupnir should synchronize policy lists immediately after startup. +# Whether or not Draupnir should synchronize policy lists immediately after startup. # Equivalent to running '!draupnir sync'. syncOnStartup: true -# Whether or not draupnir should check moderation permissions in all protected rooms on startup. +# Whether or not Draupnir should check moderation permissions in all protected rooms on startup. # Equivalent to running `!draupnir verify`. verifyPermissionsOnStartup: true -# Whether or not draupnir should actually apply bans and policy lists, +# Whether or not Draupnir should actually apply bans and policy lists, # turn on to trial some untrusted configuration or lists. noop: false -# Whether draupnir should check member lists quicker (by using a different endpoint), +# Whether Draupnir should check member lists quicker (by using a different endpoint), # keep in mind that enabling this will miss invited (but not joined) users. # # Turn on if your bot is in (very) large rooms, or in large amounts of rooms. @@ -94,7 +94,7 @@ automaticallyRedactForReasons: - "spam" - "advertising" -# A list of rooms to protect. draupnir will add this to the list it knows from its account data. +# A list of rooms to protect. Draupnir will add this to the list it knows from its account data. # # It won't, however, add it to the account data. # Manually add the room via '!draupnir rooms add' to have it stay protected regardless if this config value changes. @@ -116,21 +116,21 @@ automaticallyRedactForReasons: # Explicitly add these rooms as a protected room list if you want them protected. protectAllJoinedRooms: false -# Increase this delay to have Mjölnir wait longer between two consecutive backgrounded +# Increase this delay to have Draupnir wait longer between two consecutive backgrounded # operations. The total duration of operations will be longer, but the homeserver won't -# be affected as much. Conversely, decrease this delay to have Mjölnir chain operations +# be affected as much. Conversely, decrease this delay to have Draupnir chain operations # faster. The total duration of operations will generally be shorter, but the performance # of the homeserver may be more impacted. backgroundDelayMS: 500 -# Server administration commands, these commands will only work if draupnir is +# Server administration commands, these commands will only work if Draupnir is # a global server administrator, and the bot's server is a Synapse instance. -#admin: -# # Whether or not draupnir can temporarily take control of any eligible account from the local homeserver who's in the room -# # (with enough permissions) to "make" a user an admin. -# # -# # This only works if a local user with enough admin permissions is present in the room. -# enableMakeRoomAdminCommand: false +admin: + # Whether or not Draupnir can temporarily take control of any eligible account from the local homeserver who's in the room + # (with enough permissions) to "make" a user an admin. + # + # This only works if a local user with enough admin permissions is present in the room. + enableMakeRoomAdminCommand: true # Misc options for command handling and commands commands: @@ -138,20 +138,28 @@ commands: # # If `true`, will allow commands like `!ban`, `!help`, etc. # - # Note: draupnir can also be pinged by display name instead of having to use + # Note: Draupnir can also be pinged by display name instead of having to use # the !draupnir prefix. For example, "my_moderator_bot: ban @spammer:example.org" # will address only my_moderator_bot. allowNoPrefix: false - # Any additional bot prefixes that draupnir will listen to. i.e. adding `mod` will allow `!mod help`. + # Any additional bot prefixes that Draupnir will listen to. i.e. adding `mod` will allow `!mod help`. additionalPrefixes: - - "draupnir_bot" + - "draupnir-bot" - "draupnir" # Whether or not commands with a wildcard (*) will require an additional `--force` argument # in the command to be able to be submitted. confirmWildcardBan: true + # The default reasons to be prompted with if the reason is missing from a ban command. + ban: + defaultReasons: + - "spam" + - "brigading" + - "harassment" + - "disagreement" + # Configuration specific to certain toggle-able protections #protections: # # Configuration for the wordlist plugin, which can ban users based if they say certain From 65e7aa2406dbd3fb2f981aa8c14fd210c9850416 Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Mon, 20 Feb 2023 22:22:10 +0100 Subject: [PATCH 195/382] Update Draupnir from v1.80.0 to v1.80.1 --- roles/custom/matrix-bot-draupnir/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-draupnir/defaults/main.yml b/roles/custom/matrix-bot-draupnir/defaults/main.yml index 5900f21b..92abc7ab 100644 --- a/roles/custom/matrix-bot-draupnir/defaults/main.yml +++ b/roles/custom/matrix-bot-draupnir/defaults/main.yml @@ -4,7 +4,7 @@ matrix_bot_draupnir_enabled: true -matrix_bot_draupnir_version: "v1.80.0" +matrix_bot_draupnir_version: "v1.80.1" matrix_bot_draupnir_container_image_self_build: false matrix_bot_draupnir_container_image_self_build_repo: "https://github.com/Gnuxie/Draupnir.git" From 1c10a7727c82028b5b997ebe328d19db17d52b96 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 21 Feb 2023 07:40:49 +0200 Subject: [PATCH 196/382] Upgrade ddclient (v3.10.0-ls106 -> v3.10.0-ls110) --- roles/custom/matrix-dynamic-dns/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-dynamic-dns/defaults/main.yml b/roles/custom/matrix-dynamic-dns/defaults/main.yml index 95e197f3..4ec034a9 100644 --- a/roles/custom/matrix-dynamic-dns/defaults/main.yml +++ b/roles/custom/matrix-dynamic-dns/defaults/main.yml @@ -7,7 +7,7 @@ matrix_dynamic_dns_enabled: true # The dynamic dns daemon interval matrix_dynamic_dns_daemon_interval: '300' -matrix_dynamic_dns_version: v3.10.0-ls106 +matrix_dynamic_dns_version: v3.10.0-ls110 # The docker container to use when in mode matrix_dynamic_dns_docker_image: "{{ matrix_dynamic_dns_docker_image_name_prefix }}linuxserver/ddclient:{{ matrix_dynamic_dns_version }}" From cbc8f0c6e60e626eef493934b8a5d73317122ba1 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 21 Feb 2023 10:14:53 +0000 Subject: [PATCH 197/382] Update borgmatic 1.7.6 -> 1.7.7 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 67a1efc2..17acfa32 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git - version: v1.2.3-1.7.6-0 + version: v1.2.3-1.7.7-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git version: v9.3.6-2 From 620247673e9fec7d3ebb99af4f2665e6cbbf72ab Mon Sep 17 00:00:00 2001 From: Igor Goldenberg Date: Wed, 22 Feb 2023 18:04:13 +0500 Subject: [PATCH 198/382] Allow configure default country code in element welcome page. --- roles/custom/matrix-client-element/defaults/main.yml | 3 +++ roles/custom/matrix-client-element/templates/config.json.j2 | 1 + 2 files changed, 4 insertions(+) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index c9540ae0..60b52d60 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -191,6 +191,9 @@ matrix_client_element_self_check_validate_certificates: true # don't show the registration button on welcome page matrix_client_element_registration_enabled: false +# Default country code on welcome page when login by phone number +matrix_client_element_default_country_code: "GB" + # Controls whether presence will be enabled matrix_client_element_enable_presence_by_hs_url: ~ diff --git a/roles/custom/matrix-client-element/templates/config.json.j2 b/roles/custom/matrix-client-element/templates/config.json.j2 index 1ae16f41..e0dc993c 100644 --- a/roles/custom/matrix-client-element/templates/config.json.j2 +++ b/roles/custom/matrix-client-element/templates/config.json.j2 @@ -12,6 +12,7 @@ "custom_themes": {{ matrix_client_element_settingDefaults_custom_themes|to_json }} }, "default_theme": {{ matrix_client_element_default_theme | string|to_json }}, + "default_country_code": {{ matrix_client_element_default_country_code | string|to_json }}, "permalinkPrefix": {{ matrix_client_element_permalinkPrefix | string|to_json }}, "disable_custom_urls": {{ matrix_client_element_disable_custom_urls|to_json }}, "disable_guests": {{ matrix_client_element_disable_guests|to_json }}, From 4caeab911703e453a8dd3593620f94a44ccf7077 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Feb 2023 15:54:14 +0200 Subject: [PATCH 199/382] Fix spacing around Jinja2 filters --- .../templates/config.json.j2 | 48 +++++++++---------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/roles/custom/matrix-client-element/templates/config.json.j2 b/roles/custom/matrix-client-element/templates/config.json.j2 index e0dc993c..6f2498fd 100644 --- a/roles/custom/matrix-client-element/templates/config.json.j2 +++ b/roles/custom/matrix-client-element/templates/config.json.j2 @@ -1,49 +1,49 @@ { "default_server_config": { "m.homeserver": { - "base_url": {{ matrix_client_element_default_hs_url | string|to_json }}, - "server_name": {{ matrix_client_element_default_server_name | string|to_json }} + "base_url": {{ matrix_client_element_default_hs_url | string | to_json }}, + "server_name": {{ matrix_client_element_default_server_name | string | to_json }} }, "m.identity_server": { - "base_url": {{ matrix_client_element_default_is_url | string|to_json }} + "base_url": {{ matrix_client_element_default_is_url | string | to_json }} } }, "settingDefaults": { - "custom_themes": {{ matrix_client_element_settingDefaults_custom_themes|to_json }} + "custom_themes": {{ matrix_client_element_settingDefaults_custom_themes | to_json }} }, - "default_theme": {{ matrix_client_element_default_theme | string|to_json }}, - "default_country_code": {{ matrix_client_element_default_country_code | string|to_json }}, - "permalinkPrefix": {{ matrix_client_element_permalinkPrefix | string|to_json }}, - "disable_custom_urls": {{ matrix_client_element_disable_custom_urls|to_json }}, - "disable_guests": {{ matrix_client_element_disable_guests|to_json }}, - "brand": {{ matrix_client_element_brand|to_json }}, - "integrations_ui_url": {{ matrix_client_element_integrations_ui_url | string|to_json }}, - "integrations_rest_url": {{ matrix_client_element_integrations_rest_url | string|to_json }}, - "integrations_widgets_urls": {{ matrix_client_element_integrations_widgets_urls|to_json }}, - "integrations_jitsi_widget_url": {{ matrix_client_element_integrations_jitsi_widget_url | string|to_json }}, - "bug_report_endpoint_url": {{ matrix_client_element_bug_report_endpoint_url|to_json }}, - "showLabsSettings": {{ matrix_client_element_showLabsSettings|to_json }}, + "default_theme": {{ matrix_client_element_default_theme | string | to_json }}, + "default_country_code": {{ matrix_client_element_default_country_code | string | to_json }}, + "permalinkPrefix": {{ matrix_client_element_permalinkPrefix | string | to_json }}, + "disable_custom_urls": {{ matrix_client_element_disable_custom_urls | to_json }}, + "disable_guests": {{ matrix_client_element_disable_guests | to_json }}, + "brand": {{ matrix_client_element_brand | to_json }}, + "integrations_ui_url": {{ matrix_client_element_integrations_ui_url | string | to_json }}, + "integrations_rest_url": {{ matrix_client_element_integrations_rest_url | string | to_json }}, + "integrations_widgets_urls": {{ matrix_client_element_integrations_widgets_urls | to_json }}, + "integrations_jitsi_widget_url": {{ matrix_client_element_integrations_jitsi_widget_url | string | to_json }}, + "bug_report_endpoint_url": {{ matrix_client_element_bug_report_endpoint_url | to_json }}, + "showLabsSettings": {{ matrix_client_element_showLabsSettings | to_json }}, "roomDirectory": { - "servers": {{ matrix_client_element_roomdir_servers|to_json }} + "servers": {{ matrix_client_element_roomdir_servers | to_json }} }, - "welcomeUserId": {{ matrix_client_element_welcome_user_id|to_json }}, + "welcomeUserId": {{ matrix_client_element_welcome_user_id | to_json }}, {% if matrix_client_element_enable_presence_by_hs_url is not none %} - "enable_presence_by_hs_url": {{ matrix_client_element_enable_presence_by_hs_url|to_json }}, + "enable_presence_by_hs_url": {{ matrix_client_element_enable_presence_by_hs_url | to_json }}, {% endif %} "embeddedPages": { - "homeUrl": {{ matrix_client_element_embedded_pages_home_url | string|to_json }} + "homeUrl": {{ matrix_client_element_embedded_pages_home_url | string | to_json }} }, {% if matrix_client_element_jitsi_preferredDomain %} "jitsi": { - "preferredDomain": {{ matrix_client_element_jitsi_preferredDomain|to_json }} + "preferredDomain": {{ matrix_client_element_jitsi_preferredDomain | to_json }} }, {% endif %} {% if matrix_client_element_location_sharing_enabled %} "map_style_url": "https://{{ matrix_server_fqn_element }}/map_style.json", {% endif %} "branding": { - "authFooterLinks": {{ matrix_client_element_branding_authFooterLinks|to_json }}, - "authHeaderLogoUrl": {{ matrix_client_element_branding_authHeaderLogoUrl|to_json }}, - "welcomeBackgroundUrl": {{ matrix_client_element_branding_welcomeBackgroundUrl|to_json }} + "authFooterLinks": {{ matrix_client_element_branding_authFooterLinks | to_json }}, + "authHeaderLogoUrl": {{ matrix_client_element_branding_authHeaderLogoUrl | to_json }}, + "welcomeBackgroundUrl": {{ matrix_client_element_branding_welcomeBackgroundUrl | to_json }} } } From 643acfcb5b671d6fcd9e305c26e6fd0f1e7f4d3c Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Feb 2023 08:18:21 +0200 Subject: [PATCH 200/382] Make synapse-s3-storage-provider periodic migration schedule customizable Larger deployments may wish to run migration more often. --- roles/custom/matrix-synapse/defaults/main.yml | 3 +++ .../matrix-synapse-s3-storage-provider-migrate.timer.j2 | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index e4e2e5e2..dd69facd 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -874,6 +874,9 @@ matrix_synapse_ext_synapse_s3_storage_provider_config_threadpool_size: 40 # It specifies how old files need to have been inactive to be eligible for migration from the local filesystem to the S3 data store. # By default, we use `0` which says "all files are eligible for migration". matrix_synapse_ext_synapse_s3_storage_provider_update_db_day_count: 0 +# Specifies how often periodic migration (`matrix-synapse-s3-storage-provider-migrate.timer`) will run. +# This is a systemd timer OnCalendar definition. Learn more here: https://man.archlinux.org/man/systemd.time.7#CALENDAR_EVENTS +matrix_synapse_ext_synapse_s3_storage_provider_periodic_migration_schedule: '*-*-* 05:00:00' matrix_s3_media_store_enabled: false matrix_s3_media_store_custom_endpoint_enabled: false diff --git a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/systemd/matrix-synapse-s3-storage-provider-migrate.timer.j2 b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/systemd/matrix-synapse-s3-storage-provider-migrate.timer.j2 index 5013c7a8..14473dd7 100644 --- a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/systemd/matrix-synapse-s3-storage-provider-migrate.timer.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/systemd/matrix-synapse-s3-storage-provider-migrate.timer.j2 @@ -3,7 +3,7 @@ Description=Migrates locally-stored Synapse media store files to S3 [Timer] Unit=matrix-synapse-s3-storage-provider-migrate.service -OnCalendar=*-*-* 05:00:00 +OnCalendar={{ matrix_synapse_ext_synapse_s3_storage_provider_periodic_migration_schedule }} [Install] WantedBy=timers.target From 84c5b44bea7612b307ddae902c9f54d21d4f78d5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Feb 2023 16:09:16 +0200 Subject: [PATCH 201/382] Upgrade com.devture.ansible.role.docker_sdk_for_python --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 17acfa32..3b117ef6 100644 --- a/requirements.yml +++ b/requirements.yml @@ -4,7 +4,7 @@ version: 6.1.0 - src: git+https://github.com/devture/com.devture.ansible.role.docker_sdk_for_python.git - version: 6ba3be490b6f4c6f35ea109aeb8e533fa231b3a5 + version: 54d6d1831f063e4a99b66c722f49053d59f8ee60 - src: git+https://github.com/devture/com.devture.ansible.role.playbook_help.git version: c1f40e82b4d6b072b6f0e885239322bdaaaf554f From 0af6e6d9d970715ae1513a5ccf47f5321298fea9 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 23 Feb 2023 08:26:35 +0200 Subject: [PATCH 202/382] Make wait time for matrix-user-creator configurable and increase default (7 -> 10 sec.) --- roles/custom/matrix-user-creator/defaults/main.yml | 3 +++ roles/custom/matrix-user-creator/tasks/setup.yml | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-user-creator/defaults/main.yml b/roles/custom/matrix-user-creator/defaults/main.yml index 5c90a1e0..20aad3d1 100644 --- a/roles/custom/matrix-user-creator/defaults/main.yml +++ b/roles/custom/matrix-user-creator/defaults/main.yml @@ -20,3 +20,6 @@ matrix_user_creator_users_auto: [] # Add your own users here to have them end up in `matrix_user_creator_users` (see `vars/main.yml`) and get automatically created. # For example syntax for this variable, see the documentation for `matrix_user_creator_users` in `vars/main.yml`. matrix_user_creator_users_additional: [] + +# matrix_user_creator_homeserver_start_wait_time_seconds specifies how long to wait after starting the homeserver service and before doing our work. +matrix_user_creator_homeserver_start_wait_time_seconds: 10 diff --git a/roles/custom/matrix-user-creator/tasks/setup.yml b/roles/custom/matrix-user-creator/tasks/setup.yml index 73dfd454..d53726bd 100644 --- a/roles/custom/matrix-user-creator/tasks/setup.yml +++ b/roles/custom/matrix-user-creator/tasks/setup.yml @@ -21,7 +21,7 @@ - name: Wait a while, so that the homeserver can manage to start before creating Matrix users ansible.builtin.pause: - seconds: 7 + seconds: "{{ matrix_user_creator_homeserver_start_wait_time_seconds }}" when: matrix_user_registrator_homeserver_start_result.changed | bool - name: Ensure Matrix users are created From c8022c1a18219c281909a950c656454c959e91cd Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 23 Feb 2023 08:35:45 +0200 Subject: [PATCH 203/382] Remove useless explicit daemon reload in matrix-user-creator The next task asks for a daemon reload, so there's no need to do it twice. --- roles/custom/matrix-user-creator/tasks/setup.yml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/roles/custom/matrix-user-creator/tasks/setup.yml b/roles/custom/matrix-user-creator/tasks/setup.yml index d53726bd..bcf2fd65 100644 --- a/roles/custom/matrix-user-creator/tasks/setup.yml +++ b/roles/custom/matrix-user-creator/tasks/setup.yml @@ -8,10 +8,6 @@ # Suppress logging to avoid dumping the credentials to the shell no_log: true -- name: Ensure systemd is reloaded before starting the homeserver - ansible.builtin.service: - daemon_reload: true - - name: Ensure homeserver is started before creating Matrix users ansible.builtin.service: name: "matrix-{{ matrix_homeserver_implementation }}.service" From 40f037b36d47d222997d2a90daf90ca910e787f6 Mon Sep 17 00:00:00 2001 From: Benjamin Kampmann Date: Mon, 20 Feb 2023 22:34:16 +0100 Subject: [PATCH 204/382] Add rageshake server --- group_vars/matrix_servers | 20 ++++ playbooks/matrix.yml | 1 + roles/custom/matrix-base/defaults/main.yml | 3 + .../matrix-nginx-proxy/defaults/main.yml | 12 +++ .../tasks/setup_nginx_proxy.yml | 13 +++ .../matrix-nginx-proxy/templates/labels.j2 | 11 ++ .../nginx/conf.d/matrix-rageshake.conf.j2 | 100 ++++++++++++++++++ .../custom/matrix-rageshake/defaults/main.yml | 38 +++++++ roles/custom/matrix-rageshake/tasks/main.yml | 19 ++++ .../matrix-rageshake/tasks/setup_install.yml | 66 ++++++++++++ .../tasks/setup_uninstall.yml | 24 +++++ .../tasks/validate_config.yml | 1 + .../matrix-rageshake/templates/config.j2 | 2 + .../systemd/matrix-rageshake.service.j2 | 41 +++++++ 14 files changed, 351 insertions(+) create mode 100644 roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-rageshake.conf.j2 create mode 100644 roles/custom/matrix-rageshake/defaults/main.yml create mode 100644 roles/custom/matrix-rageshake/tasks/main.yml create mode 100644 roles/custom/matrix-rageshake/tasks/setup_install.yml create mode 100644 roles/custom/matrix-rageshake/tasks/setup_uninstall.yml create mode 100644 roles/custom/matrix-rageshake/tasks/validate_config.yml create mode 100644 roles/custom/matrix-rageshake/templates/config.j2 create mode 100644 roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index f148e042..1307679d 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -296,6 +296,8 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-coturn.service', 'priority': 4000, 'groups': ['matrix', 'coturn']}] if matrix_coturn_enabled else []) + + ([{'name': 'matrix-rageshake.service', 'priority': 4000, 'groups': ['matrix', 'rageshake']}] if matrix_rageshake_enabled else []) + + ([{'name': 'matrix-coturn-reload.timer', 'priority': 5000, 'groups': ['matrix', 'coturn']}] if (matrix_coturn_enabled and matrix_coturn_tls_enabled) else []) + ([{'name': 'matrix-dimension.service', 'priority': 4000, 'groups': ['matrix', 'integration-managers', 'dimension']}] if matrix_dimension_enabled else []) @@ -1995,6 +1997,18 @@ matrix_corporal_matrix_registration_shared_secret: "{{ matrix_synapse_registrati # ###################################################################### + +###################################################################### +# +# /matrix-rageshake +# +###################################################################### + +# We don't enable rageshake by default. +matrix_rageshake_enabled: false + +matrix_rageshake_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm64'] }}" + ###################################################################### # # matrix-coturn @@ -2383,6 +2397,7 @@ matrix_nginx_proxy_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled an matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" +matrix_nginx_proxy_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled }}" matrix_nginx_proxy_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" @@ -2400,6 +2415,7 @@ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_ent matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" +matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" @@ -2483,6 +2499,8 @@ matrix_nginx_proxy_systemd_wanted_services_list: | + (['matrix-dimension.service'] if matrix_dimension_enabled else []) + + (['matrix-rageshake.service'] if matrix_rageshake_enabled else []) + + (['matrix-sygnal.service'] if matrix_sygnal_enabled else []) + ([(ntfy_identifier + '.service')] if ntfy_enabled else []) @@ -2524,6 +2542,8 @@ matrix_ssl_domains_to_obtain_certificates_for: | + ([ntfy_hostname] if ntfy_enabled else []) + + ([matrix_server_fqn_rageshake] if matrix_rageshake_enabled else []) + + (matrix_bot_postmoogle_domains if matrix_bot_postmoogle_enabled else []) + ([matrix_domain] if matrix_nginx_proxy_base_domain_serving_enabled else []) diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 6980a6f9..a425cd88 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -82,6 +82,7 @@ - custom/matrix-bot-draupnir - custom/matrix-bot-chatgpt - custom/matrix-cactus-comments + - custom/matrix-rageshake - custom/matrix-synapse - custom/matrix-synapse-reverse-proxy-companion - custom/matrix-dendrite diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index da5459f8..0f2e3b48 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -90,6 +90,9 @@ matrix_server_fqn_sygnal: "sygnal.{{ matrix_domain }}" # This is where you access the ntfy push notification service. matrix_server_fqn_ntfy: "ntfy.{{ matrix_domain }}" +# This is where you access rageshake. +matrix_server_fqn_rageshake: "rageshake.{{ matrix_domain }}" + matrix_federation_public_port: 8448 # The name of the Traefik entrypoint for handling Matrix Federation diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index b656f074..b5915dc4 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -71,6 +71,11 @@ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname: "{{ matrix matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" matrix_nginx_proxy_container_labels_traefik_proxy_dimension_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname }}`)" +matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_enabled: false +matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_hostname: "{{ matrix_server_fqn_rageshake }}" +matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" +matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_hostname }}`)" + matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_hostname: "{{ matrix_server_fqn_etherpad }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" @@ -257,6 +262,10 @@ matrix_nginx_proxy_proxy_matrix_federation_port: 8448 matrix_nginx_proxy_proxy_dimension_enabled: false matrix_nginx_proxy_proxy_dimension_hostname: "{{ matrix_server_fqn_dimension }}" +# Controls whether proxying the rageshake domain should be done. +matrix_nginx_proxy_proxy_rageshake_enabled: false +matrix_nginx_proxy_proxy_rageshake_hostname: "{{ matrix_server_fqn_rageshake }}" + # Controls whether proxying the etherpad domain should be done. matrix_nginx_proxy_proxy_etherpad_enabled: false matrix_nginx_proxy_proxy_etherpad_hostname: "{{ matrix_server_fqn_etherpad }}" @@ -444,6 +453,9 @@ matrix_nginx_proxy_proxy_buscarron_additional_server_configuration_blocks: [] # A list of strings containing additional configuration blocks to add to Dimension's server configuration (matrix-dimension.conf). matrix_nginx_proxy_proxy_dimension_additional_server_configuration_blocks: [] +# A list of strings containing additional configuration blocks to add to Rageshake's server configuration (matrix-rageshake.conf). +matrix_nginx_proxy_proxy_rageshake_additional_server_configuration_blocks: [] + # A list of strings containing additional configuration blocks to add to etherpad's server configuration (matrix-etherpad.conf). matrix_nginx_proxy_proxy_etherpad_additional_server_configuration_blocks: [] diff --git a/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml b/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml index 5aa63597..1a55e28f 100644 --- a/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml +++ b/roles/custom/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml @@ -129,6 +129,13 @@ mode: 0644 when: matrix_nginx_proxy_proxy_dimension_enabled | bool +- name: Ensure Matrix nginx-proxy configuration for rageshake domain exists + ansible.builtin.template: + src: "{{ role_path }}/templates/nginx/conf.d/matrix-rageshake.conf.j2" + dest: "{{ matrix_nginx_proxy_confd_path }}/matrix-rageshake.conf" + mode: 0644 + when: matrix_nginx_proxy_proxy_rageshake_enabled | bool + - name: Ensure Matrix nginx-proxy configuration for etherpad domain exists ansible.builtin.template: src: "{{ role_path }}/templates/nginx/conf.d/matrix-etherpad.conf.j2" @@ -291,6 +298,12 @@ state: absent when: "not matrix_nginx_proxy_proxy_dimension_enabled | bool" +- name: Ensure Matrix nginx-proxy configuration for rageshake domain deleted + ansible.builtin.file: + path: "{{ matrix_nginx_proxy_confd_path }}/matrix-rageshake.conf" + state: absent + when: "not matrix_nginx_proxy_proxy_rageshake_enabled | bool" + - name: Ensure Matrix nginx-proxy configuration for goneb domain deleted ansible.builtin.file: path: "{{ matrix_nginx_proxy_confd_path }}/matrix-bot-go-neb.conf" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 1e60ba6d..00c9c40a 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -62,6 +62,17 @@ traefik.http.routers.matrix-nginx-proxy-dimension.entrypoints={{ matrix_nginx_pr {% endif %} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_enabled %} +# Rageshake +traefik.http.routers.matrix-nginx-proxy-rageshake.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_rule }} +traefik.http.routers.matrix-nginx-proxy-rageshake.service=matrix-nginx-proxy-web +traefik.http.routers.matrix-nginx-proxy-rageshake.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_tls | to_json }} +{% if matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_tls %} +traefik.http.routers.matrix-nginx-proxy-rageshake.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} +{% endif %} +traefik.http.routers.matrix-nginx-proxy-rageshake.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} +{% endif %} + {% if matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled %} # Etherpad traefik.http.routers.matrix-nginx-proxy-etherpad.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_rule }} diff --git a/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-rageshake.conf.j2 b/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-rageshake.conf.j2 new file mode 100644 index 00000000..5da96684 --- /dev/null +++ b/roles/custom/matrix-nginx-proxy/templates/nginx/conf.d/matrix-rageshake.conf.j2 @@ -0,0 +1,100 @@ +#jinja2: lstrip_blocks: "True" + +{% macro render_vhost_directives() %} + gzip on; + gzip_types text/plain application/json application/javascript text/css image/x-icon font/ttf image/gif; + {% if matrix_nginx_proxy_hsts_preload_enabled %} + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always; + {% else %} + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + {% endif %} + add_header X-XSS-Protection "{{ matrix_nginx_proxy_xss_protection }}"; + add_header X-Content-Type-Options nosniff; + {% if matrix_nginx_proxy_floc_optout_enabled %} + add_header Permissions-Policy interest-cohort=() always; + {% endif %} + +{% for configuration_block in matrix_nginx_proxy_proxy_rageshake_additional_server_configuration_blocks %} + {{- configuration_block }} +{% endfor %} + + location / { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s; + set $backend "matrix-rageshake:9110"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://127.0.0.1:9110; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For {{ matrix_nginx_proxy_x_forwarded_for }}; + } +{% endmacro %} + +server { + listen {{ 8080 if matrix_nginx_proxy_enabled else 80 }}; + listen [::]:{{ 8080 if matrix_nginx_proxy_enabled else 80 }}; + + server_name {{ matrix_nginx_proxy_proxy_rageshake_hostname }}; + + server_tokens off; + root /dev/null; + + {% if matrix_nginx_proxy_https_enabled %} + location /.well-known/acme-challenge { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver {{ matrix_nginx_proxy_http_level_resolver }} valid=5s; + set $backend "matrix-certbot:8080"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://127.0.0.1:{{ matrix_ssl_lets_encrypt_certbot_standalone_http_port }}; + {% endif %} + } + + location / { + return 301 https://$http_host$request_uri; + } + {% else %} + {{ render_vhost_directives() }} + {% endif %} +} + +{% if matrix_nginx_proxy_https_enabled %} +server { + listen {{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; + listen [::]:{{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; + + server_name {{ matrix_nginx_proxy_proxy_rageshake_hostname }}; + + server_tokens off; + root /dev/null; + + ssl_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_rageshake_hostname }}/fullchain.pem; + ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_rageshake_hostname }}/privkey.pem; + + ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }}; + {% if matrix_nginx_proxy_ssl_ciphers != '' %} + ssl_ciphers {{ matrix_nginx_proxy_ssl_ciphers }}; + {% endif %} + ssl_prefer_server_ciphers {{ matrix_nginx_proxy_ssl_prefer_server_ciphers }}; + + {% if matrix_nginx_proxy_ocsp_stapling_enabled %} + ssl_stapling on; + ssl_stapling_verify on; + ssl_trusted_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_rageshake_hostname }}/chain.pem; + {% endif %} + + {% if matrix_nginx_proxy_ssl_session_tickets_off %} + ssl_session_tickets off; + {% endif %} + ssl_session_cache {{ matrix_nginx_proxy_ssl_session_cache }}; + ssl_session_timeout {{ matrix_nginx_proxy_ssl_session_timeout }}; + + {{ render_vhost_directives() }} +} +{% endif %} diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml new file mode 100644 index 00000000..743b3000 --- /dev/null +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -0,0 +1,38 @@ +--- +# rageshake is a issue submission service from matrix.org +# Project source code URL: https://github.com/matrix-org/rageshake/ + +matrix_rageshake_enabled: false + +matrix_rageshake_container_image_self_build: false +matrix_rageshake_docker_repo: "https://github.com/matrix-org/rageshake/" +matrix_rageshake_docker_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" +matrix_rageshake_docker_src_files_path: "{{ matrix_base_data_path }}/Dockerfile" + +matrix_rageshake_version: "master" +matrix_rageshake_docker_image: "{{ matrix_rageshake_docker_image_name_prefix }}matrix-org/rageshake:{{ matrix_rageshake_version }}" +matrix_rageshake_docker_image_name_prefix: "{{ 'localhost/' if matrix_rageshake_container_image_self_build else 'ghcr.io/' }}" +matrix_rageshake_docker_image_force_pull: "{{ matrix_rageshake_docker_image.endswith(':master') }}" + +matrix_rageshake_base_path: "{{ matrix_base_data_path }}/rageshake" +matrix_rageshake_config_path: "{{ matrix_rageshake_base_path }}/config" +matrix_rageshake_data_path: "{{ matrix_rageshake_base_path }}/data" + +# A list of extra arguments to pass to the container +matrix_rageshake_container_extra_arguments: [] + +# List of systemd services that matrix-rageshake.service depends on +matrix_rageshake_systemd_required_services_list: ["docker.service"] + +# List of systemd services that matrix-rageshake.service wants +matrix_rageshake_systemd_wanted_services_list: [] + +# Rageshake Settings + +# Additional config to pass to the rageshake. +# See https://github.com/matrix-org/rageshake/blob/master/rageshake.sample.yaml +# +# Example: +# matrix_rageshake_config: | +# github_token: secrettoken +matrix_rageshake_config: "" diff --git a/roles/custom/matrix-rageshake/tasks/main.yml b/roles/custom/matrix-rageshake/tasks/main.yml new file mode 100644 index 00000000..f0e03e66 --- /dev/null +++ b/roles/custom/matrix-rageshake/tasks/main.yml @@ -0,0 +1,19 @@ +--- +- block: + - when: matrix_rageshake_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_rageshake_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + tags: + - setup-all + - setup-rageshake + - install-all + - install-rageshake + +- block: + - when: not matrix_rageshake_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + tags: + - setup-all + - setup-rageshake diff --git a/roles/custom/matrix-rageshake/tasks/setup_install.yml b/roles/custom/matrix-rageshake/tasks/setup_install.yml new file mode 100644 index 00000000..b081960e --- /dev/null +++ b/roles/custom/matrix-rageshake/tasks/setup_install.yml @@ -0,0 +1,66 @@ +--- +- name: Ensure rageshake paths exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - { path: "{{ matrix_rageshake_config_path }}", when: true } + - { path: "{{ matrix_rageshake_data_path }}", when: true } + - { + path: "{{ matrix_rageshake_docker_src_files_path }}", + when: matrix_rageshake_container_image_self_build, + } + when: "item.when | bool" + +- name: Ensure rageshake config file created + ansible.builtin.template: + src: "{{ role_path }}/templates/config.j2" + dest: "{{ matrix_rageshake_config_path }}/config.yml" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + mode: 0640 + +- name: Ensure rageshake image is pulled + community.docker.docker_image: + name: "{{ matrix_rageshake_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_rageshake_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_rageshake_docker_image_force_pull }}" + when: "not matrix_rageshake_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure rageshake repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_rageshake_docker_repo }}" + version: "{{ matrix_rageshake_docker_repo_version }}" + dest: "{{ matrix_rageshake_docker_src_files_path }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_rageshake_git_pull_results + when: "matrix_rageshake_container_image_self_build | bool" + +- name: Ensure rageshake image is built + community.docker.docker_image: + name: "{{ matrix_rageshake_docker_image }}" + source: build + force_source: "{{ matrix_rageshake_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_rageshake_docker_src_files_path }}" + pull: true + when: "matrix_rageshake_container_image_self_build | bool" + +- name: Ensure matrix-rageshake.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-rageshake.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-rageshake.service" + mode: 0644 + register: matrix_rageshake_systemd_service_result diff --git a/roles/custom/matrix-rageshake/tasks/setup_uninstall.yml b/roles/custom/matrix-rageshake/tasks/setup_uninstall.yml new file mode 100644 index 00000000..133ac76d --- /dev/null +++ b/roles/custom/matrix-rageshake/tasks/setup_uninstall.yml @@ -0,0 +1,24 @@ +--- +- name: Check existence of matrix-rageshake service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-rageshake.service" + register: matrix_rageshake_service_stat + +- when: matrix_rageshake_service_stat.stat.exists | bool + block: + - name: Ensure matrix-rageshake is stopped + ansible.builtin.service: + name: matrix-rageshake + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-rageshake.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-rageshake.service" + state: absent + + - name: Ensure Matrix rageshake paths don't exist + ansible.builtin.file: + path: "{{ matrix_rageshake_base_path }}" + state: absent diff --git a/roles/custom/matrix-rageshake/tasks/validate_config.yml b/roles/custom/matrix-rageshake/tasks/validate_config.yml new file mode 100644 index 00000000..ed97d539 --- /dev/null +++ b/roles/custom/matrix-rageshake/tasks/validate_config.yml @@ -0,0 +1 @@ +--- diff --git a/roles/custom/matrix-rageshake/templates/config.j2 b/roles/custom/matrix-rageshake/templates/config.j2 new file mode 100644 index 00000000..2b0d64b7 --- /dev/null +++ b/roles/custom/matrix-rageshake/templates/config.j2 @@ -0,0 +1,2 @@ +api_prefix: "https://{{ matrix_server_fqn_rageshake }}/api/" +{{ matrix_rageshake_config | to_nice_yaml(indent=2, width=999999) }} diff --git a/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 b/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 new file mode 100644 index 00000000..6e818e0f --- /dev/null +++ b/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 @@ -0,0 +1,41 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Rageshake Service +{% for service in matrix_rageshake_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_rageshake_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-rageshake 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-rageshake 2>/dev/null || true' + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-rageshake \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --tmpfs /tmp \ + --network={{ matrix_docker_network }} \ + --mount type=bind,src={{ matrix_rageshake_config_path }},dst=/config \ + --mount type=bind,src={{ matrix_rageshake_data_path }},dst=/bugs \ + {% for arg in matrix_rageshake_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_rageshake_docker_image }} \ + --config /config/config.yml + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-rageshake 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-rageshake 2>/dev/null || true' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-rageshake + +[Install] +WantedBy=multi-user.target From 53f8a0c6ae8d00f365bf3793660a5ca30f0ef601 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 24 Feb 2023 20:27:15 +0200 Subject: [PATCH 205/382] Upgrade com.devture.ansible.role.traefik --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 3b117ef6..cbdd37ca 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.7-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: da90b6cdbbb9374ca72c1b530867add2fd8f024e + version: 5e3248cd4d17db9ff3ba90904e7330150dab599e - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 From c139d139a5c12a091fa170b1b67afcb64505de8f Mon Sep 17 00:00:00 2001 From: Tupsi Date: Sat, 25 Feb 2023 00:46:54 +0100 Subject: [PATCH 206/382] Update configuring-playbook-own-webserver.md own traefik server option must have the path to the acme.json. Otherwise the service will not start. --- docs/configuring-playbook-own-webserver.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 34e0bf3e..22170f40 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -33,6 +33,8 @@ This will install Traefik in the place of `matrix-nginx-proxy`. Traefik will man matrix_playbook_reverse_proxy_type: other-traefik-container matrix_playbook_reverse_proxyable_services_additional_network: your-traefik-network + +devture_traefik_certs_dumper_ssl_dir_path: "/path/to/your/traefiks/acme.json/directory" ``` In this mode all roles will still have Traefik labels attached. You will, however, need to configure your Traefik instance and its entrypoints. From 55f43dcc6d6f6733ce2346112c13b43778080108 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:09:23 +0200 Subject: [PATCH 207/382] Fixup matrix-rageshake section in group vars --- group_vars/matrix_servers | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 1307679d..4669b331 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2000,7 +2000,7 @@ matrix_corporal_matrix_registration_shared_secret: "{{ matrix_synapse_registrati ###################################################################### # -# /matrix-rageshake +# matrix-rageshake # ###################################################################### @@ -2009,6 +2009,12 @@ matrix_rageshake_enabled: false matrix_rageshake_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm64'] }}" +###################################################################### +# +# /matrix-rageshake +# +###################################################################### + ###################################################################### # # matrix-coturn From d513c0f446589dc34249b8ce417f4a1d21263ca8 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:20:09 +0200 Subject: [PATCH 208/382] Document why the rageshake version is master --- roles/custom/matrix-rageshake/defaults/main.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 743b3000..16a936bc 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -9,6 +9,8 @@ matrix_rageshake_docker_repo: "https://github.com/matrix-org/rageshake/" matrix_rageshake_docker_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" matrix_rageshake_docker_src_files_path: "{{ matrix_base_data_path }}/Dockerfile" +# There are no stable container image tags yet. +# See: https://github.com/matrix-org/rageshake/issues/69 matrix_rageshake_version: "master" matrix_rageshake_docker_image: "{{ matrix_rageshake_docker_image_name_prefix }}matrix-org/rageshake:{{ matrix_rageshake_version }}" matrix_rageshake_docker_image_name_prefix: "{{ 'localhost/' if matrix_rageshake_container_image_self_build else 'ghcr.io/' }}" From 19e0b8374bab3de8ba4ec4dad12b7bcc9b907690 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:20:25 +0200 Subject: [PATCH 209/382] Enable matrix_rageshake_enabled in the role by default We disable it via `group_vars/matrix_servers`, as we do for all other roles. --- roles/custom/matrix-rageshake/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 16a936bc..3568d23f 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -2,7 +2,7 @@ # rageshake is a issue submission service from matrix.org # Project source code URL: https://github.com/matrix-org/rageshake/ -matrix_rageshake_enabled: false +matrix_rageshake_enabled: true matrix_rageshake_container_image_self_build: false matrix_rageshake_docker_repo: "https://github.com/matrix-org/rageshake/" From db0104978f97e540db418eeb3b4b7cdcba602c5a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:23:07 +0200 Subject: [PATCH 210/382] Extract api_prefix rageshake configuration into matrix_rageshake_config_api_prefix --- roles/custom/matrix-rageshake/defaults/main.yml | 2 ++ roles/custom/matrix-rageshake/templates/config.j2 | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 3568d23f..4f574d3f 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -29,6 +29,8 @@ matrix_rageshake_systemd_required_services_list: ["docker.service"] # List of systemd services that matrix-rageshake.service wants matrix_rageshake_systemd_wanted_services_list: [] +matrix_rageshake_config_api_prefix: "https://{{ matrix_server_fqn_rageshake }}/api/" + # Rageshake Settings # Additional config to pass to the rageshake. diff --git a/roles/custom/matrix-rageshake/templates/config.j2 b/roles/custom/matrix-rageshake/templates/config.j2 index 2b0d64b7..08c90438 100644 --- a/roles/custom/matrix-rageshake/templates/config.j2 +++ b/roles/custom/matrix-rageshake/templates/config.j2 @@ -1,2 +1,2 @@ -api_prefix: "https://{{ matrix_server_fqn_rageshake }}/api/" +api_prefix: {{ matrix_rageshake_config_api_prefix | to_json }} {{ matrix_rageshake_config | to_nice_yaml(indent=2, width=999999) }} From a51c9521a8f976d2778be3a45d411682fa60b073 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:23:45 +0200 Subject: [PATCH 211/382] Rename some task files --- .../matrix-rageshake/tasks/{setup_install.yml => install.yml} | 0 roles/custom/matrix-rageshake/tasks/main.yml | 4 ++-- .../tasks/{setup_uninstall.yml => uninstall.yml} | 0 3 files changed, 2 insertions(+), 2 deletions(-) rename roles/custom/matrix-rageshake/tasks/{setup_install.yml => install.yml} (100%) rename roles/custom/matrix-rageshake/tasks/{setup_uninstall.yml => uninstall.yml} (100%) diff --git a/roles/custom/matrix-rageshake/tasks/setup_install.yml b/roles/custom/matrix-rageshake/tasks/install.yml similarity index 100% rename from roles/custom/matrix-rageshake/tasks/setup_install.yml rename to roles/custom/matrix-rageshake/tasks/install.yml diff --git a/roles/custom/matrix-rageshake/tasks/main.yml b/roles/custom/matrix-rageshake/tasks/main.yml index f0e03e66..c330964a 100644 --- a/roles/custom/matrix-rageshake/tasks/main.yml +++ b/roles/custom/matrix-rageshake/tasks/main.yml @@ -4,7 +4,7 @@ ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_rageshake_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" tags: - setup-all - setup-rageshake @@ -13,7 +13,7 @@ - block: - when: not matrix_rageshake_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" tags: - setup-all - setup-rageshake diff --git a/roles/custom/matrix-rageshake/tasks/setup_uninstall.yml b/roles/custom/matrix-rageshake/tasks/uninstall.yml similarity index 100% rename from roles/custom/matrix-rageshake/tasks/setup_uninstall.yml rename to roles/custom/matrix-rageshake/tasks/uninstall.yml From 8339ebdf5894ef6e2b12627fe9097672cb06cfcb Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:29:22 +0200 Subject: [PATCH 212/382] Redo the way matryx-rageshake configuration is done This makes it consistent with the rest of the playbook: - there's a default config which has various variables controlling settings - there's also an `_extension_yaml` variable, which lets you override it --- .../custom/matrix-rageshake/defaults/main.yml | 33 ++++++++++++++----- .../custom/matrix-rageshake/tasks/install.yml | 5 ++- .../matrix-rageshake/templates/config.j2 | 2 -- .../matrix-rageshake/templates/config.yml.j2 | 4 +++ 4 files changed, 31 insertions(+), 13 deletions(-) delete mode 100644 roles/custom/matrix-rageshake/templates/config.j2 create mode 100644 roles/custom/matrix-rageshake/templates/config.yml.j2 diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 4f574d3f..a1bbe242 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -31,12 +31,29 @@ matrix_rageshake_systemd_wanted_services_list: [] matrix_rageshake_config_api_prefix: "https://{{ matrix_server_fqn_rageshake }}/api/" -# Rageshake Settings - -# Additional config to pass to the rageshake. -# See https://github.com/matrix-org/rageshake/blob/master/rageshake.sample.yaml +# Default Rageshake configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. # -# Example: -# matrix_rageshake_config: | -# github_token: secrettoken -matrix_rageshake_config: "" +# For a more advanced customization, you can extend the default (see `matrix_rageshake_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_rageshake_configuration_yaml: "{{ lookup('template', 'templates/config.yml.j2') }}" + +matrix_rageshake_configuration_extension_yaml: | + # Your custom YAML configuration for Synapse goes here. + # This configuration extends the default starting configuration (`matrix_rageshake_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_rageshake_configuration_yaml`. + # + # Example configuration extension follows: + # + # github_project_mappings: + # my-app: octocat/HelloWorld + +matrix_rageshake_configuration_extension: "{{ matrix_rageshake_configuration_extension_yaml | from_yaml if matrix_rageshake_configuration_extension_yaml | from_yaml is mapping else {} }}" + +# Holds the final Synapse configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_rageshake_configuration_yaml`. +matrix_rageshake_configuration: "{{ matrix_rageshake_configuration_yaml | from_yaml | combine(matrix_rageshake_configuration_extension, recursive=True) }}" diff --git a/roles/custom/matrix-rageshake/tasks/install.yml b/roles/custom/matrix-rageshake/tasks/install.yml index b081960e..3f4d64c5 100644 --- a/roles/custom/matrix-rageshake/tasks/install.yml +++ b/roles/custom/matrix-rageshake/tasks/install.yml @@ -16,8 +16,8 @@ when: "item.when | bool" - name: Ensure rageshake config file created - ansible.builtin.template: - src: "{{ role_path }}/templates/config.j2" + ansible.builtin.copy: + content: "{{ matrix_rageshake_configuration | to_nice_yaml(indent=2, width=999999) }}" dest: "{{ matrix_rageshake_config_path }}/config.yml" owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" @@ -63,4 +63,3 @@ src: "{{ role_path }}/templates/systemd/matrix-rageshake.service.j2" dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-rageshake.service" mode: 0644 - register: matrix_rageshake_systemd_service_result diff --git a/roles/custom/matrix-rageshake/templates/config.j2 b/roles/custom/matrix-rageshake/templates/config.j2 deleted file mode 100644 index 08c90438..00000000 --- a/roles/custom/matrix-rageshake/templates/config.j2 +++ /dev/null @@ -1,2 +0,0 @@ -api_prefix: {{ matrix_rageshake_config_api_prefix | to_json }} -{{ matrix_rageshake_config | to_nice_yaml(indent=2, width=999999) }} diff --git a/roles/custom/matrix-rageshake/templates/config.yml.j2 b/roles/custom/matrix-rageshake/templates/config.yml.j2 new file mode 100644 index 00000000..6b0c2664 --- /dev/null +++ b/roles/custom/matrix-rageshake/templates/config.yml.j2 @@ -0,0 +1,4 @@ +# Default configuration for Rageshake. +# To extend it, use `matrix_rageshake_configuration_extension_yaml`. + +api_prefix: {{ matrix_rageshake_config_api_prefix | to_json }} From 306679103b28ce6642f390e99fe264ab73483706 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:32:19 +0200 Subject: [PATCH 213/382] Require self-building of rageshake for arm64 There are no arm64 images published.. yet --- group_vars/matrix_servers | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 4669b331..2ad11473 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2007,7 +2007,7 @@ matrix_corporal_matrix_registration_shared_secret: "{{ matrix_synapse_registrati # We don't enable rageshake by default. matrix_rageshake_enabled: false -matrix_rageshake_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm64'] }}" +matrix_rageshake_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" ###################################################################### # From 5934de84be6721ecac0187cf473abcd058a5b455 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 12:59:56 +0200 Subject: [PATCH 214/382] Reorder matrix-rageshake default variables a bit --- roles/custom/matrix-rageshake/defaults/main.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index a1bbe242..85a3e50e 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -4,11 +4,6 @@ matrix_rageshake_enabled: true -matrix_rageshake_container_image_self_build: false -matrix_rageshake_docker_repo: "https://github.com/matrix-org/rageshake/" -matrix_rageshake_docker_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" -matrix_rageshake_docker_src_files_path: "{{ matrix_base_data_path }}/Dockerfile" - # There are no stable container image tags yet. # See: https://github.com/matrix-org/rageshake/issues/69 matrix_rageshake_version: "master" @@ -20,6 +15,11 @@ matrix_rageshake_base_path: "{{ matrix_base_data_path }}/rageshake" matrix_rageshake_config_path: "{{ matrix_rageshake_base_path }}/config" matrix_rageshake_data_path: "{{ matrix_rageshake_base_path }}/data" +matrix_rageshake_container_image_self_build: false +matrix_rageshake_docker_repo: "https://github.com/matrix-org/rageshake/" +matrix_rageshake_docker_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" +matrix_rageshake_docker_src_files_path: "{{ matrix_base_data_path }}/Dockerfile" + # A list of extra arguments to pass to the container matrix_rageshake_container_extra_arguments: [] From 9a8ebf3d8c5e96d9703abb7b681e5852196eebbe Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 13:01:11 +0200 Subject: [PATCH 215/382] matrix_rageshake_docker_ -> matrix_rageshake_container_ --- .../custom/matrix-rageshake/defaults/main.yml | 12 ++++++------ .../custom/matrix-rageshake/tasks/install.yml | 18 +++++++++--------- .../systemd/matrix-rageshake.service.j2 | 2 +- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 85a3e50e..e29ca62f 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -7,18 +7,18 @@ matrix_rageshake_enabled: true # There are no stable container image tags yet. # See: https://github.com/matrix-org/rageshake/issues/69 matrix_rageshake_version: "master" -matrix_rageshake_docker_image: "{{ matrix_rageshake_docker_image_name_prefix }}matrix-org/rageshake:{{ matrix_rageshake_version }}" -matrix_rageshake_docker_image_name_prefix: "{{ 'localhost/' if matrix_rageshake_container_image_self_build else 'ghcr.io/' }}" -matrix_rageshake_docker_image_force_pull: "{{ matrix_rageshake_docker_image.endswith(':master') }}" +matrix_rageshake_container_image: "{{ matrix_rageshake_container_image_name_prefix }}matrix-org/rageshake:{{ matrix_rageshake_version }}" +matrix_rageshake_container_image_name_prefix: "{{ 'localhost/' if matrix_rageshake_container_image_self_build else 'ghcr.io/' }}" +matrix_rageshake_container_image_force_pull: "{{ matrix_rageshake_container_image.endswith(':master') }}" matrix_rageshake_base_path: "{{ matrix_base_data_path }}/rageshake" matrix_rageshake_config_path: "{{ matrix_rageshake_base_path }}/config" matrix_rageshake_data_path: "{{ matrix_rageshake_base_path }}/data" matrix_rageshake_container_image_self_build: false -matrix_rageshake_docker_repo: "https://github.com/matrix-org/rageshake/" -matrix_rageshake_docker_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" -matrix_rageshake_docker_src_files_path: "{{ matrix_base_data_path }}/Dockerfile" +matrix_rageshake_container_repo: "https://github.com/matrix-org/rageshake/" +matrix_rageshake_container_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" +matrix_rageshake_container_src_files_path: "{{ matrix_base_data_path }}/Dockerfile" # A list of extra arguments to pass to the container matrix_rageshake_container_extra_arguments: [] diff --git a/roles/custom/matrix-rageshake/tasks/install.yml b/roles/custom/matrix-rageshake/tasks/install.yml index 3f4d64c5..6b75b969 100644 --- a/roles/custom/matrix-rageshake/tasks/install.yml +++ b/roles/custom/matrix-rageshake/tasks/install.yml @@ -10,7 +10,7 @@ - { path: "{{ matrix_rageshake_config_path }}", when: true } - { path: "{{ matrix_rageshake_data_path }}", when: true } - { - path: "{{ matrix_rageshake_docker_src_files_path }}", + path: "{{ matrix_rageshake_container_src_files_path }}", when: matrix_rageshake_container_image_self_build, } when: "item.when | bool" @@ -25,10 +25,10 @@ - name: Ensure rageshake image is pulled community.docker.docker_image: - name: "{{ matrix_rageshake_docker_image }}" + name: "{{ matrix_rageshake_container_image }}" source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_rageshake_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_rageshake_docker_image_force_pull }}" + force_source: "{{ matrix_rageshake_container_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_rageshake_container_image_force_pull }}" when: "not matrix_rageshake_container_image_self_build | bool" register: result retries: "{{ devture_playbook_help_container_retries_count }}" @@ -37,9 +37,9 @@ - name: Ensure rageshake repository is present on self-build ansible.builtin.git: - repo: "{{ matrix_rageshake_docker_repo }}" - version: "{{ matrix_rageshake_docker_repo_version }}" - dest: "{{ matrix_rageshake_docker_src_files_path }}" + repo: "{{ matrix_rageshake_container_repo }}" + version: "{{ matrix_rageshake_container_repo_version }}" + dest: "{{ matrix_rageshake_container_src_files_path }}" force: "yes" become: true become_user: "{{ matrix_user_username }}" @@ -48,13 +48,13 @@ - name: Ensure rageshake image is built community.docker.docker_image: - name: "{{ matrix_rageshake_docker_image }}" + name: "{{ matrix_rageshake_container_image }}" source: build force_source: "{{ matrix_rageshake_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" build: dockerfile: Dockerfile - path: "{{ matrix_rageshake_docker_src_files_path }}" + path: "{{ matrix_rageshake_container_src_files_path }}" pull: true when: "matrix_rageshake_container_image_self_build | bool" diff --git a/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 b/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 index 6e818e0f..5e6cfece 100644 --- a/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 +++ b/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 @@ -28,7 +28,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% for arg in matrix_rageshake_container_extra_arguments %} {{ arg }} \ {% endfor %} - {{ matrix_rageshake_docker_image }} \ + {{ matrix_rageshake_container_image }} \ --config /config/config.yml ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-rageshake 2>/dev/null || true' From 39be2ef71abf6d85c73381b5f22cfae292db5fe1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 13:01:38 +0200 Subject: [PATCH 216/382] Fix matrix_rageshake_container_src_files_path value --- roles/custom/matrix-rageshake/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index e29ca62f..5be3a1ca 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -18,7 +18,7 @@ matrix_rageshake_data_path: "{{ matrix_rageshake_base_path }}/data" matrix_rageshake_container_image_self_build: false matrix_rageshake_container_repo: "https://github.com/matrix-org/rageshake/" matrix_rageshake_container_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" -matrix_rageshake_container_src_files_path: "{{ matrix_base_data_path }}/Dockerfile" +matrix_rageshake_container_src_files_path: "{{ matrix_rageshake_base_path }}/container-src" # A list of extra arguments to pass to the container matrix_rageshake_container_extra_arguments: [] From 4c64e092d00e0abc84b9695bc32c852ea7d17771 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 13:09:53 +0200 Subject: [PATCH 217/382] Fix rageshake self-building BuildX is required for building this container image. --- .../custom/matrix-rageshake/tasks/install.yml | 31 +++++++++---------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/roles/custom/matrix-rageshake/tasks/install.yml b/roles/custom/matrix-rageshake/tasks/install.yml index 6b75b969..a7b6a5b1 100644 --- a/roles/custom/matrix-rageshake/tasks/install.yml +++ b/roles/custom/matrix-rageshake/tasks/install.yml @@ -7,12 +7,12 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" with_items: - - { path: "{{ matrix_rageshake_config_path }}", when: true } - - { path: "{{ matrix_rageshake_data_path }}", when: true } - - { - path: "{{ matrix_rageshake_container_src_files_path }}", - when: matrix_rageshake_container_image_self_build, - } + - path: "{{ matrix_rageshake_config_path }}" + when: true + - path: "{{ matrix_rageshake_data_path }}" + when: true + - path: "{{ matrix_rageshake_container_src_files_path }}" + when: matrix_rageshake_container_image_self_build | bool when: "item.when | bool" - name: Ensure rageshake config file created @@ -46,17 +46,14 @@ register: matrix_rageshake_git_pull_results when: "matrix_rageshake_container_image_self_build | bool" -- name: Ensure rageshake image is built - community.docker.docker_image: - name: "{{ matrix_rageshake_container_image }}" - source: build - force_source: "{{ matrix_rageshake_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" - build: - dockerfile: Dockerfile - path: "{{ matrix_rageshake_container_src_files_path }}" - pull: true - when: "matrix_rageshake_container_image_self_build | bool" +- name: Ensure rageshake container image is built + ansible.builtin.command: + cmd: |- + {{ devture_systemd_docker_base_host_command_docker }} buildx build + --tag={{ matrix_rageshake_container_image }} + --file={{ matrix_rageshake_container_src_files_path }}/Dockerfile + {{ matrix_rageshake_container_src_files_path }} + when: matrix_rageshake_container_image_self_build | bool - name: Ensure matrix-rageshake.service installed ansible.builtin.template: From 233e25326457fdbb6c65e74412c6f0f824a310f9 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 13:15:34 +0200 Subject: [PATCH 218/382] Add native Traefik support to rageshake --- group_vars/matrix_servers | 15 +++- .../matrix-nginx-proxy/defaults/main.yml | 5 -- .../matrix-nginx-proxy/templates/labels.j2 | 11 --- .../custom/matrix-rageshake/defaults/main.yml | 69 +++++++++++++++++-- .../custom/matrix-rageshake/tasks/install.yml | 18 ++++- .../matrix-rageshake/tasks/uninstall.yml | 3 +- .../tasks/validate_config.yml | 30 ++++++++ .../matrix-rageshake/templates/labels.j2 | 45 ++++++++++++ .../systemd/matrix-rageshake.service.j2 | 14 +++- 9 files changed, 180 insertions(+), 30 deletions(-) create mode 100644 roles/custom/matrix-rageshake/templates/labels.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 2ad11473..dc24392e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2009,6 +2009,18 @@ matrix_rageshake_enabled: false matrix_rageshake_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" +matrix_rageshake_hostname: "{{ matrix_server_fqn_rageshake }}" + +matrix_rageshake_container_network: "{{ matrix_docker_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-rageshake' }}" + +matrix_rageshake_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + +matrix_rageshake_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9110') if matrix_playbook_service_host_bind_interface_prefix else '' }}" + +matrix_rageshake_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +matrix_rageshake_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_rageshake_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" + ###################################################################### # # /matrix-rageshake @@ -2403,7 +2415,7 @@ matrix_nginx_proxy_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled an matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" -matrix_nginx_proxy_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled }}" +matrix_nginx_proxy_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" @@ -2421,7 +2433,6 @@ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_ent matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" -matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index b5915dc4..061d8f93 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -71,11 +71,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname: "{{ matrix matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" matrix_nginx_proxy_container_labels_traefik_proxy_dimension_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname }}`)" -matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_hostname: "{{ matrix_server_fqn_rageshake }}" -matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_hostname: "{{ matrix_server_fqn_etherpad }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 00c9c40a..1e60ba6d 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -62,17 +62,6 @@ traefik.http.routers.matrix-nginx-proxy-dimension.entrypoints={{ matrix_nginx_pr {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_enabled %} -# Rageshake -traefik.http.routers.matrix-nginx-proxy-rageshake.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_rule }} -traefik.http.routers.matrix-nginx-proxy-rageshake.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-rageshake.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_rageshake_tls %} -traefik.http.routers.matrix-nginx-proxy-rageshake.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-rageshake.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - {% if matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled %} # Etherpad traefik.http.routers.matrix-nginx-proxy-etherpad.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_rule }} diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 5be3a1ca..456838ed 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -4,21 +4,76 @@ matrix_rageshake_enabled: true +matrix_rageshake_scheme: https + +# The hostname at which rageshake is served. +matrix_rageshake_hostname: '' + +# The path at which rageshake is exposed. +# This value must either be `/` or not end with a slash (e.g. `/rageshake`). +matrix_rageshake_path_prefix: / + # There are no stable container image tags yet. # See: https://github.com/matrix-org/rageshake/issues/69 matrix_rageshake_version: "master" -matrix_rageshake_container_image: "{{ matrix_rageshake_container_image_name_prefix }}matrix-org/rageshake:{{ matrix_rageshake_version }}" -matrix_rageshake_container_image_name_prefix: "{{ 'localhost/' if matrix_rageshake_container_image_self_build else 'ghcr.io/' }}" -matrix_rageshake_container_image_force_pull: "{{ matrix_rageshake_container_image.endswith(':master') }}" matrix_rageshake_base_path: "{{ matrix_base_data_path }}/rageshake" matrix_rageshake_config_path: "{{ matrix_rageshake_base_path }}/config" matrix_rageshake_data_path: "{{ matrix_rageshake_base_path }}/data" +matrix_rageshake_container_src_files_path: "{{ matrix_rageshake_base_path }}/container-src" + +matrix_rageshake_container_image: "{{ matrix_rageshake_container_image_name_prefix }}matrix-org/rageshake:{{ matrix_rageshake_container_image_tag }}" +matrix_rageshake_container_image_name_prefix: "{{ 'localhost/' if matrix_rageshake_container_image_self_build else matrix_rageshake_container_registry_prefix }}" +matrix_rageshake_container_image_force_pull: "{{ matrix_rageshake_container_image.endswith(':master') }}" +matrix_rageshake_container_image_tag: "{{ matrix_rageshake_version }}" +matrix_rageshake_container_registry_prefix: ghcr.io/ matrix_rageshake_container_image_self_build: false -matrix_rageshake_container_repo: "https://github.com/matrix-org/rageshake/" -matrix_rageshake_container_repo_version: "{{ 'master' if matrix_rageshake_version == 'master' else matrix_rageshake_version }}" -matrix_rageshake_container_src_files_path: "{{ matrix_rageshake_base_path }}/container-src" +matrix_rageshake_container_image_self_build_repo: "https://github.com/matrix-org/rageshake/" +matrix_rageshake_container_image_self_build_repo_version: "{{ matrix_rageshake_version }}" + +# Controls whether the container exposes its HTTP port (tcp/9110 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:9110"), or empty string to not expose. +matrix_rageshake_container_http_host_bind_port: '' + +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_rageshake_container_network: matrix-rageshake + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_rageshake_container_additional_networks: [] + +# matrix_rageshake_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_rageshake_container_labels_additional_labels`. +matrix_rageshake_container_labels_traefik_enabled: true +matrix_rageshake_container_labels_traefik_docker_network: "{{ matrix_rageshake_container_network }}" +matrix_rageshake_container_labels_traefik_hostname: "{{ matrix_rageshake_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/rageshake`). +matrix_rageshake_container_labels_traefik_path_prefix: "{{ matrix_rageshake_path_prefix }}" +matrix_rageshake_container_labels_traefik_rule: "Host(`{{ matrix_rageshake_container_labels_traefik_hostname }}`){% if matrix_rageshake_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_rageshake_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_rageshake_container_labels_traefik_priority: 0 +matrix_rageshake_container_labels_traefik_entrypoints: web-secure +matrix_rageshake_container_labels_traefik_tls: "{{ matrix_rageshake_container_labels_traefik_entrypoints != 'web' }}" +matrix_rageshake_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_rageshake_container_labels_traefik_additional_response_headers_custom` +matrix_rageshake_container_labels_traefik_additional_response_headers: "{{ matrix_rageshake_container_labels_traefik_additional_response_headers_auto | combine(matrix_rageshake_container_labels_traefik_additional_response_headers_custom) }}" +matrix_rageshake_container_labels_traefik_additional_response_headers_auto: {} +matrix_rageshake_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_rageshake_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_rageshake_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_rageshake_container_labels_additional_labels: '' # A list of extra arguments to pass to the container matrix_rageshake_container_extra_arguments: [] @@ -29,7 +84,7 @@ matrix_rageshake_systemd_required_services_list: ["docker.service"] # List of systemd services that matrix-rageshake.service wants matrix_rageshake_systemd_wanted_services_list: [] -matrix_rageshake_config_api_prefix: "https://{{ matrix_server_fqn_rageshake }}/api/" +matrix_rageshake_config_api_prefix: "{{ matrix_rageshake_scheme }}://{{ matrix_rageshake_hostname }}{{ matrix_rageshake_path_prefix }}{{ '' if matrix_rageshake_path_prefix == '/' else '/' }}api/" # Default Rageshake configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. diff --git a/roles/custom/matrix-rageshake/tasks/install.yml b/roles/custom/matrix-rageshake/tasks/install.yml index a7b6a5b1..f9461680 100644 --- a/roles/custom/matrix-rageshake/tasks/install.yml +++ b/roles/custom/matrix-rageshake/tasks/install.yml @@ -1,4 +1,5 @@ --- + - name: Ensure rageshake paths exist ansible.builtin.file: path: "{{ item.path }}" @@ -23,6 +24,14 @@ group: "{{ matrix_user_groupname }}" mode: 0640 +- name: Ensure rageshake labels installed + ansible.builtin.template: + src: "{{ role_path }}/templates/labels.j2" + dest: "{{ matrix_rageshake_base_path }}/labels" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + - name: Ensure rageshake image is pulled community.docker.docker_image: name: "{{ matrix_rageshake_container_image }}" @@ -37,8 +46,8 @@ - name: Ensure rageshake repository is present on self-build ansible.builtin.git: - repo: "{{ matrix_rageshake_container_repo }}" - version: "{{ matrix_rageshake_container_repo_version }}" + repo: "{{ matrix_rageshake_container_image_self_build_repo }}" + version: "{{ matrix_rageshake_container_image_self_build_version }}" dest: "{{ matrix_rageshake_container_src_files_path }}" force: "yes" become: true @@ -55,6 +64,11 @@ {{ matrix_rageshake_container_src_files_path }} when: matrix_rageshake_container_image_self_build | bool +- name: Ensure rageshake container network is created + community.general.docker_network: + name: "{{ matrix_rageshake_container_network }}" + driver: bridge + - name: Ensure matrix-rageshake.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-rageshake.service.j2" diff --git a/roles/custom/matrix-rageshake/tasks/uninstall.yml b/roles/custom/matrix-rageshake/tasks/uninstall.yml index 133ac76d..8f04d989 100644 --- a/roles/custom/matrix-rageshake/tasks/uninstall.yml +++ b/roles/custom/matrix-rageshake/tasks/uninstall.yml @@ -1,4 +1,5 @@ --- + - name: Check existence of matrix-rageshake service ansible.builtin.stat: path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-rageshake.service" @@ -18,7 +19,7 @@ path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-rageshake.service" state: absent - - name: Ensure Matrix rageshake paths don't exist + - name: Ensure rageshake paths don't exist ansible.builtin.file: path: "{{ matrix_rageshake_base_path }}" state: absent diff --git a/roles/custom/matrix-rageshake/tasks/validate_config.yml b/roles/custom/matrix-rageshake/tasks/validate_config.yml index ed97d539..b61a486d 100644 --- a/roles/custom/matrix-rageshake/tasks/validate_config.yml +++ b/roles/custom/matrix-rageshake/tasks/validate_config.yml @@ -1 +1,31 @@ --- + +- name: Fail if required rageshake settings not defined + ansible.builtin.fail: + msg: > + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_rageshake_hostname + - matrix_rageshake_path_prefix + - matrix_rageshake_container_network + +- when: matrix_rageshake_container_labels_traefik_enabled | bool + block: + - name: Fail if required rageshake Traefik settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_rageshake_container_labels_traefik_hostname + - matrix_rageshake_container_labels_traefik_path_prefix + + # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). + # Knowing that `matrix_rageshake_container_labels_traefik_path_prefix` does not end with a slash + # ensures we know how to set these routes up without having to do "does it end with a slash" checks elsewhere. + - name: Fail if matrix_rageshake_container_labels_traefik_path_prefix ends with a slash + ansible.builtin.fail: + msg: >- + matrix_rageshake_container_labels_traefik_path_prefix (`{{ matrix_rageshake_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/rageshake`). + when: "matrix_rageshake_container_labels_traefik_path_prefix != '/' and matrix_rageshake_container_labels_traefik_path_prefix[-1] == '/'" diff --git a/roles/custom/matrix-rageshake/templates/labels.j2 b/roles/custom/matrix-rageshake/templates/labels.j2 new file mode 100644 index 00000000..cdaf5f6a --- /dev/null +++ b/roles/custom/matrix-rageshake/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_rageshake_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_rageshake_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_rageshake_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_rageshake_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-rageshake-slashless-redirect.redirectregex.regex=({{ matrix_rageshake_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-rageshake-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-rageshake-slashless-redirect'] %} +{% endif %} + +{% if matrix_rageshake_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-rageshake-strip-prefix.stripprefix.prefixes={{ matrix_rageshake_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-rageshake-strip-prefix'] %} +{% endif %} + +{% if matrix_rageshake_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_rageshake_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-rageshake-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-rageshake-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-rageshake.rule={{ matrix_rageshake_container_labels_traefik_rule }} +{% if matrix_rageshake_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-rageshake.priority={{ matrix_rageshake_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-rageshake.service=matrix-rageshake +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-rageshake.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-rageshake.entrypoints={{ matrix_rageshake_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-rageshake.tls={{ matrix_rageshake_container_labels_traefik_tls | to_json }} +{% if matrix_rageshake_container_labels_traefik_tls %} +traefik.http.routers.matrix-rageshake.tls.certResolver={{ matrix_rageshake_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-rageshake.loadbalancer.server.port=9110 +{% endif %} + +{{ matrix_rageshake_container_labels_additional_labels }} diff --git a/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 b/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 index 5e6cfece..75006c2d 100644 --- a/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 +++ b/roles/custom/matrix-rageshake/templates/systemd/matrix-rageshake.service.j2 @@ -16,23 +16,33 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-rageshake 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-rageshake 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-rageshake \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-rageshake \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ --tmpfs /tmp \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_rageshake_container_network }} \ --mount type=bind,src={{ matrix_rageshake_config_path }},dst=/config \ --mount type=bind,src={{ matrix_rageshake_data_path }},dst=/bugs \ + --label-file={{ matrix_rageshake_base_path }}/labels \ {% for arg in matrix_rageshake_container_extra_arguments %} {{ arg }} \ {% endfor %} {{ matrix_rageshake_container_image }} \ --config /config/config.yml +{% for network in matrix_rageshake_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-rageshake +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-rageshake + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-rageshake 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-rageshake 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-rageshake From b5e9ff59266eb2a255140337af2c5499b8a75afd Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 14:00:42 +0200 Subject: [PATCH 219/382] Add Rageshake documentation --- docs/configuring-playbook-rageshake.md | 65 ++++++++++++++++++++++++++ docs/configuring-playbook.md | 2 + 2 files changed, 67 insertions(+) create mode 100644 docs/configuring-playbook-rageshake.md diff --git a/docs/configuring-playbook-rageshake.md b/docs/configuring-playbook-rageshake.md new file mode 100644 index 00000000..274d0c8b --- /dev/null +++ b/docs/configuring-playbook-rageshake.md @@ -0,0 +1,65 @@ +# Setting up Rageshake (optional) + +The playbook can install and configure the [rageshake](https://github.com/matrix-org/rageshake) bug report server for you. + +This is useful if you're developing your own applications and would like to collect bug reports for them. + + +## Decide on a domain and path + +By default, Rageshake is configured to use its own dedicated domain (`rageshake.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). + +You can override the domain and path like this: + +```yaml +# Switch to the domain used for Matrix services (`matrix.DOMAIN`), +# so we won't need to an additional DNS records for Rageshake. +matrix_rageshake_hostname: "{{ matrix_server_fqn_matrix }}" + +# Expose under the /rageshake subpath +matrix_rageshake_path_prefix: /rageshake +``` + +**NOTE**: When using `matrix-nginx-proxy` instead of Traefik, you won't be able to override the path prefix. You can only override the domain, but that needs to happen using another variable: `matrix_server_fqn_rageshake` (e.g. `matrix_server_fqn_rageshake: "some-domain.{{ matrix_domain }}"`). + + +## Adjusting DNS records + +Once you've decided on the domain and path, **you may need to adjust your DNS** records to point the Rageshake domain to the Matrix server. + +If you've decided to reuse the `matrix.` domain, you won't need to do any extra DNS configuration. + + +## Enabling the Rageshake service + +Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file (adapt to your needs): + +```yaml +matrix_rageshake_enabled: true +``` + +Rageshake has various options which don't have dedicated Ansible variables. You can see the full list of options in the [`rageshake.sample.yaml` file](https://github.com/matrix-org/rageshake/blob/master/rageshake.sample.yaml). + +To set these, you can make use of the `matrix_rageshake_configuration_extension_yaml` variable like this: + +```yaml +matrix_rageshake_configuration_extension_yaml: | + github_token: secrettoken + + github_project_mappings: + my-app: octocat/HelloWorld +``` + + +## Installing + +After configuring the playbook, run the [installation](installing.md) command again: + +``` +ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start +``` + + +## Usage + +Refer to the [rageshake documentation](https://github.com/matrix-org/rageshake) for available APIs, etc. diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index 1f3cb7a4..c94977a2 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -198,3 +198,5 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up the ntfy push notifications server](configuring-playbook-ntfy.md) (optional) - [Setting up a Cactus Comments server](configuring-playbook-cactus-comments.md) - a federated comment system built on Matrix (optional) + +- [Setting up the Rageshake bug report server](configuring-playbook-rageshake.md) (optional) From a45c70fd66aa8ea08d5fe416c6b06d192ef75211 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 14:04:20 +0200 Subject: [PATCH 220/382] Announce Rageshake support --- CHANGELOG.md | 9 +++++++++ README.md | 5 +++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5dd5e9f6..a49de417 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,12 @@ +# 2023-02-25 + +## Rageshake support + +Thanks to [Benjamin Kampmann](https://github.com/gnunicorn), the playbook can now install and configure the [Rageshake](https://github.com/matrix-org/rageshake) bug report server. + +Additional details are available in [Setting up Rageshake](docs/configuring-playbook-rageshake.md). + + # 2023-02-17 ## Synapse templates customization support diff --git a/README.md b/README.md index 174ea4c3..d284909f 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ You can always re-run the playbook later to add or remove components. The homeserver is the backbone of your matrix system. Choose one from the following list. | Name | Default? | Description | Documentation | -| ---- | -------- | ----------- | ------------- | +| ---- | -------- | ----------- | ------------- | | [Synapse](https://github.com/matrix-org/synapse) | ✓ | Storing your data and managing your presence in the [Matrix](http://matrix.org/) network | [Link](docs/configuring-playbook-synapse.md) | | [Conduit](https://conduit.rs) | x | Storing your data and managing your presence in the [Matrix](http://matrix.org/) network. Conduit is a lightweight open-source server implementation of the Matrix Specification with a focus on easy setup and low system requirements | [Link](docs/configuring-playbook-conduit.md) | | [Dendrite](https://github.com/matrix-org/dendrite) | x | Storing your data and managing your presence in the [Matrix](http://matrix.org/) network. Dendrite is a second-generation Matrix homeserver written in Go, an alternative to Synapse. | [Link](docs/configuring-playbook-dendrite.md) | @@ -66,7 +66,7 @@ Services that run on the server to make the various parts of your installation w Extend and modify how users are authenticated on your homeserver. | Name | Default? | Description | Documentation | -| ---- | -------- | ----------- | ------------- | +| ---- | -------- | ----------- | ------------- | | [matrix-synapse-rest-auth](https://github.com/ma1uta/matrix-synapse-rest-password-provider) (advanced) | x | REST authentication password provider module | [Link](docs/configuring-playbook-rest-auth.md) | |[matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth) (advanced) | x | Password provider module | [Link](docs/configuring-playbook-shared-secret-auth.md) | | [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3) (advanced) | x | LDAP Auth password provider module | [Link](configuring-playbook-ldap-auth.md) | @@ -144,6 +144,7 @@ Services that help you in administrating and monitoring your matrix installation | [synapse-admin](https://github.com/Awesome-Technologies/synapse-admin) | x | A web UI tool for administrating users and rooms on your Matrix server | [Link](docs/configuring-playbook-synapse-admin.md) | | Metrics and Graphs | x | Consists of the [Prometheus](https://prometheus.io) time-series database server, the Prometheus [node-exporter](https://prometheus.io/docs/guides/node-exporter/) host metrics exporter, and the [Grafana](https://grafana.com/) web UI | [Link](docs/configuring-playbook-prometheus-grafana.md) | | [Borg](https://borgbackup.org) | x | Backups | [Link](docs/configuring-playbook-backup-borg.md) | +| [Rageshake](https://github.com/matrix-org/rageshake) | x | Bug report server | [Link](docs/configuring-playbook-rageshake.md) | ### Misc From 6c2d397853b50751ee268341497427f59c93c765 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 14:58:41 +0200 Subject: [PATCH 221/382] Fix inaccurate Buscarron project description comment --- roles/custom/matrix-bot-buscarron/defaults/main.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-buscarron/defaults/main.yml b/roles/custom/matrix-bot-buscarron/defaults/main.yml index f887e262..c2d8d2de 100644 --- a/roles/custom/matrix-bot-buscarron/defaults/main.yml +++ b/roles/custom/matrix-bot-buscarron/defaults/main.yml @@ -1,5 +1,6 @@ --- -# buscarron is a helpdesk bot + +# buscarron is a Web forms (HTTP POST) to Matrix service. # Project source code URL: https://gitlab.com/etke.cc/buscarron matrix_bot_buscarron_enabled: true From 0220c851e8781e4468b1ff1b68e205c1e1c05212 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 15:02:30 +0200 Subject: [PATCH 222/382] Add multiple container networks support to Buscarron --- group_vars/matrix_servers | 2 ++ roles/custom/matrix-bot-buscarron/defaults/main.yml | 8 ++++++++ .../matrix-bot-buscarron/tasks/setup_install.yml | 5 +++++ .../systemd/matrix-bot-buscarron.service.j2 | 13 +++++++++++-- 4 files changed, 26 insertions(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index dc24392e..ee00dc58 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1685,6 +1685,8 @@ matrix_bot_buscarron_database_engine: "{{ 'postgres' if devture_postgres_enabled matrix_bot_buscarron_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" matrix_bot_buscarron_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'buscarron.bot.db', rounds=655555) | to_uuid }}" +matrix_bot_container_network: "{{ matrix_docker_network }}" + ###################################################################### # # /matrix-bot-buscarron diff --git a/roles/custom/matrix-bot-buscarron/defaults/main.yml b/roles/custom/matrix-bot-buscarron/defaults/main.yml index c2d8d2de..6ed6e962 100644 --- a/roles/custom/matrix-bot-buscarron/defaults/main.yml +++ b/roles/custom/matrix-bot-buscarron/defaults/main.yml @@ -20,6 +20,14 @@ matrix_bot_buscarron_config_path: "{{ matrix_bot_buscarron_base_path }}/config" matrix_bot_buscarron_data_path: "{{ matrix_bot_buscarron_base_path }}/data" matrix_bot_buscarron_data_store_path: "{{ matrix_bot_buscarron_data_path }}/store" +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_bot_container_network: matrix-bot-buscarron + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_bot_container_additional_networks: [] + # A list of extra arguments to pass to the container matrix_bot_buscarron_container_extra_arguments: [] diff --git a/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml b/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml index 6ae0f9b9..94fe93f6 100644 --- a/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml +++ b/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml @@ -89,6 +89,11 @@ dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-buscarron.service" mode: 0644 +- name: Ensure buscarron container network is created + community.general.docker_network: + name: "{{ matrix_bot_buscarron_container_network }}" + driver: bridge + - name: Ensure matrix-bot-buscarron.service restarted, if necessary ansible.builtin.service: name: "matrix-bot-buscarron.service" diff --git a/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 b/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 index 60909dc9..b3ed673d 100644 --- a/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 +++ b/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 @@ -16,12 +16,14 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-buscarron 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-buscarron 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-bot-buscarron \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-bot-buscarron \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_bot_buscarron_container_network }} \ --env-file={{ matrix_bot_buscarron_config_path }}/env \ --mount type=bind,src={{ matrix_bot_buscarron_data_path }},dst=/data \ {% for arg in matrix_bot_buscarron_container_extra_arguments %} @@ -29,8 +31,15 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endfor %} {{ matrix_bot_buscarron_docker_image }} +{% for network in matrix_bot_buscarron_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-bot-buscarron +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-bot-buscarron + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-buscarron 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-buscarron 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-bot-buscarron From 63408096cfb6ae0fd0aefc5fcd25aace68ea995e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 15:06:29 +0200 Subject: [PATCH 223/382] Fix up Buscarron description on its documentation page --- docs/configuring-playbook-bot-buscarron.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/configuring-playbook-bot-buscarron.md b/docs/configuring-playbook-bot-buscarron.md index 0b80ba40..bfee92ee 100644 --- a/docs/configuring-playbook-bot-buscarron.md +++ b/docs/configuring-playbook-bot-buscarron.md @@ -2,8 +2,7 @@ The playbook can install and configure [buscarron](https://gitlab.com/etke.cc/buscarron) for you. -It's a bot you can use to setup **your own helpdesk on matrix** -It's a bot you can use to send any form (HTTP POST, HTML) to a (encrypted) matrix room +Buscarron is bot that receives HTTP POST submissions of web forms and forwards them to a Matrix room. ## Adjusting the playbook configuration From d7578880b6124bdbe961011ccff95f6587b317f1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 15:07:47 +0200 Subject: [PATCH 224/382] Fix typo on Rageshake documentation page --- docs/configuring-playbook-rageshake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-rageshake.md b/docs/configuring-playbook-rageshake.md index 274d0c8b..fe45db26 100644 --- a/docs/configuring-playbook-rageshake.md +++ b/docs/configuring-playbook-rageshake.md @@ -13,7 +13,7 @@ You can override the domain and path like this: ```yaml # Switch to the domain used for Matrix services (`matrix.DOMAIN`), -# so we won't need to an additional DNS records for Rageshake. +# so we won't need to add additional DNS records for Rageshake. matrix_rageshake_hostname: "{{ matrix_server_fqn_matrix }}" # Expose under the /rageshake subpath From bc5dda2b3aba7842ae63127ef9cc6e8f5816d52f Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 15:10:40 +0200 Subject: [PATCH 225/382] Reorder some Buscarron default variables and fix some typos Fixes a regression introduced in 0220c851e8781e4468b1 --- group_vars/matrix_servers | 2 +- .../matrix-bot-buscarron/defaults/main.yml | 17 +++++++++-------- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index ee00dc58..6af65b3a 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1685,7 +1685,7 @@ matrix_bot_buscarron_database_engine: "{{ 'postgres' if devture_postgres_enabled matrix_bot_buscarron_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" matrix_bot_buscarron_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'buscarron.bot.db', rounds=655555) | to_uuid }}" -matrix_bot_container_network: "{{ matrix_docker_network }}" +matrix_bot_buscarron_container_network: "{{ matrix_docker_network }}" ###################################################################### # diff --git a/roles/custom/matrix-bot-buscarron/defaults/main.yml b/roles/custom/matrix-bot-buscarron/defaults/main.yml index 6ed6e962..195359dd 100644 --- a/roles/custom/matrix-bot-buscarron/defaults/main.yml +++ b/roles/custom/matrix-bot-buscarron/defaults/main.yml @@ -5,28 +5,29 @@ matrix_bot_buscarron_enabled: true +matrix_bot_buscarron_version: v1.3.1 + +matrix_bot_buscarron_base_path: "{{ matrix_base_data_path }}/buscarron" +matrix_bot_buscarron_config_path: "{{ matrix_bot_buscarron_base_path }}/config" +matrix_bot_buscarron_data_path: "{{ matrix_bot_buscarron_base_path }}/data" +matrix_bot_buscarron_data_store_path: "{{ matrix_bot_buscarron_data_path }}/store" + matrix_bot_buscarron_container_image_self_build: false matrix_bot_buscarron_docker_repo: "https://gitlab.com/etke.cc/buscarron.git" matrix_bot_buscarron_docker_repo_version: "{{ matrix_bot_buscarron_version }}" matrix_bot_buscarron_docker_src_files_path: "{{ matrix_base_data_path }}/buscarron/docker-src" -matrix_bot_buscarron_version: v1.3.1 matrix_bot_buscarron_docker_image: "{{ matrix_bot_buscarron_docker_image_name_prefix }}etke.cc/buscarron:{{ matrix_bot_buscarron_version }}" matrix_bot_buscarron_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_buscarron_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_buscarron_docker_image_force_pull: "{{ matrix_bot_buscarron_docker_image.endswith(':latest') }}" -matrix_bot_buscarron_base_path: "{{ matrix_base_data_path }}/buscarron" -matrix_bot_buscarron_config_path: "{{ matrix_bot_buscarron_base_path }}/config" -matrix_bot_buscarron_data_path: "{{ matrix_bot_buscarron_base_path }}/data" -matrix_bot_buscarron_data_store_path: "{{ matrix_bot_buscarron_data_path }}/store" - # The base container network. It will be auto-created by this role if it doesn't exist already. -matrix_bot_container_network: matrix-bot-buscarron +matrix_bot_buscarron_container_network: matrix-bot-buscarron # A list of additional container networks that the container would be connected to. # The role does not create these networks, so make sure they already exist. # Use this to expose this container to another reverse proxy, which runs in a different container network. -matrix_bot_container_additional_networks: [] +matrix_bot_buscarron_container_additional_networks: [] # A list of extra arguments to pass to the container matrix_bot_buscarron_container_extra_arguments: [] From 3852a740bc5e0dba7ff662df3ce3a0f857f0216c Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 15:50:35 +0200 Subject: [PATCH 226/382] Mention that Buscarron is aggressive when it comes to banning --- docs/configuring-playbook-bot-buscarron.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/configuring-playbook-bot-buscarron.md b/docs/configuring-playbook-bot-buscarron.md index bfee92ee..58b27db7 100644 --- a/docs/configuring-playbook-bot-buscarron.md +++ b/docs/configuring-playbook-bot-buscarron.md @@ -66,4 +66,12 @@ To use the bot, invite the `@bot.buscarron:DOMAIN` to the room you specified in ``` +**NOTE**: to fight against spam, Buscarron is **very aggressive when it comes to banning** and will ban you if: + +- if you hit the homepage (HTTP `GET` request to `/`) +- if you submit a form to the wrong URL (`POST` request to `/non-existing-form`) +- if `hasemail` is enabled for the form (like in the example above) and you don't submit an `email` field + +If you get banned, you'd need to restart the process by running the playbook with `--tags=start` or running `systemctl restart matrix-bot-buscarron` on the server. + You can also refer to the upstream [documentation](https://gitlab.com/etke.cc/buscarron). From 725b2beed7346328ca939fb3daa0a1ecfd71091d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 15:50:48 +0200 Subject: [PATCH 227/382] Add native Traefik support to Buscarron --- docs/configuring-playbook-bot-buscarron.md | 35 ++++++++++----- group_vars/matrix_servers | 23 ++++++++-- .../matrix-bot-buscarron/defaults/main.yml | 37 +++++++++++++++ .../tasks/setup_install.yml | 9 ++-- .../tasks/validate_config.yml | 1 + .../matrix-bot-buscarron/templates/labels.j2 | 45 +++++++++++++++++++ .../systemd/matrix-bot-buscarron.service.j2 | 1 + .../matrix-nginx-proxy/defaults/main.yml | 5 --- .../matrix-nginx-proxy/templates/labels.j2 | 11 ----- 9 files changed, 135 insertions(+), 32 deletions(-) create mode 100644 roles/custom/matrix-bot-buscarron/templates/labels.j2 diff --git a/docs/configuring-playbook-bot-buscarron.md b/docs/configuring-playbook-bot-buscarron.md index 58b27db7..dd577854 100644 --- a/docs/configuring-playbook-bot-buscarron.md +++ b/docs/configuring-playbook-bot-buscarron.md @@ -5,6 +5,31 @@ The playbook can install and configure [buscarron](https://gitlab.com/etke.cc/bu Buscarron is bot that receives HTTP POST submissions of web forms and forwards them to a Matrix room. +## Decide on a domain and path + +By default, Rageshake is configured to use its own dedicated domain (`buscarron.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). + +You can override the domain and path like this: + +```yaml +# Switch to the domain used for Matrix services (`matrix.DOMAIN`), +# so we won't need to add additional DNS records for Rageshake. +matrix_bot_buscarron_hostname: "{{ matrix_server_fqn_matrix }}" + +# Expose under the /buscarron subpath +matrix_bot_buscarron_path_prefix: /buscarron +``` + +**NOTE**: When using `matrix-nginx-proxy` instead of Traefik, you won't be able to override the path prefix. You can only override the domain, but that needs to happen using another variable: `matrix_server_fqn_buscarron` (e.g. `matrix_server_fqn_buscarron: "form.{{ matrix_domain }}"`). + + +## Adjusting DNS records + +Once you've decided on the domain and path, **you may need to adjust your DNS** records to point the Buscarron domain to the Matrix server. + +If you've decided to reuse the `matrix.` domain, you won't need to do any extra DNS configuration. + + ## Adjusting the playbook configuration Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file: @@ -30,16 +55,6 @@ matrix_bot_buscarron_forms: matrix_bot_buscarron_spamlist: [] # (optional) list of emails/domains/hosts (with wildcards support) that should be rejected automatically ``` -You will also need to add a DNS record so that buscarron can be accessed. -By default buscarron will use https://buscarron.DOMAIN so you will need to create an CNAME record for `buscarron`. -See [Configuring DNS](configuring-dns.md). - -If you would like to use a different domain, add the following to your configuration file (changing it to use your preferred domain): - -```yaml -matrix_server_fqn_buscarron: "form.{{ matrix_domain }}" -``` - ## Installing diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6af65b3a..afc1511c 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1685,7 +1685,25 @@ matrix_bot_buscarron_database_engine: "{{ 'postgres' if devture_postgres_enabled matrix_bot_buscarron_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" matrix_bot_buscarron_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'buscarron.bot.db', rounds=655555) | to_uuid }}" -matrix_bot_buscarron_container_network: "{{ matrix_docker_network }}" +matrix_bot_buscarron_hostname: "{{ matrix_server_fqn_buscarron }}" + +matrix_bot_buscarron_container_network: "{{ matrix_docker_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-bot-buscarron' }}" + +# For now, we attach this to the matrix-nginx-proxy network (unless that's the same network as the main one for us), +# because that's where the homeserver is expected to be accessed from. +# In the future, this will need to be redone. +matrix_bot_buscarron_container_additional_networks: | + {{ + ( + ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + + + ([matrix_nginx_proxy_container_network] if matrix_nginx_proxy_enabled and matrix_bot_buscarron_container_network != matrix_nginx_proxy_container_network else []) + ) | unique + }} + +matrix_bot_buscarron_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +matrix_bot_buscarron_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_bot_buscarron_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" ###################################################################### # @@ -2415,7 +2433,7 @@ matrix_nginx_proxy_proxy_matrix_enabled: true matrix_nginx_proxy_proxy_element_enabled: "{{ matrix_client_element_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" -matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" +matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" matrix_nginx_proxy_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" @@ -2433,7 +2451,6 @@ matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true -matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" diff --git a/roles/custom/matrix-bot-buscarron/defaults/main.yml b/roles/custom/matrix-bot-buscarron/defaults/main.yml index 195359dd..0596c440 100644 --- a/roles/custom/matrix-bot-buscarron/defaults/main.yml +++ b/roles/custom/matrix-bot-buscarron/defaults/main.yml @@ -7,6 +7,13 @@ matrix_bot_buscarron_enabled: true matrix_bot_buscarron_version: v1.3.1 +# The hostname at which Buscarron is served. +matrix_bot_buscarron_hostname: '' + +# The path at which Buscarron is exposed. +# This value must either be `/` or not end with a slash (e.g. `/buscarron`). +matrix_bot_buscarron_path_prefix: / + matrix_bot_buscarron_base_path: "{{ matrix_base_data_path }}/buscarron" matrix_bot_buscarron_config_path: "{{ matrix_bot_buscarron_base_path }}/config" matrix_bot_buscarron_data_path: "{{ matrix_bot_buscarron_base_path }}/data" @@ -29,6 +36,36 @@ matrix_bot_buscarron_container_network: matrix-bot-buscarron # Use this to expose this container to another reverse proxy, which runs in a different container network. matrix_bot_buscarron_container_additional_networks: [] +# matrix_bot_buscarron_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_bot_buscarron_container_labels_additional_labels`. +matrix_bot_buscarron_container_labels_traefik_enabled: true +matrix_bot_buscarron_container_labels_traefik_docker_network: "{{ matrix_bot_buscarron_container_network }}" +matrix_bot_buscarron_container_labels_traefik_hostname: "{{ matrix_bot_buscarron_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/buscarron`). +matrix_bot_buscarron_container_labels_traefik_path_prefix: "{{ matrix_bot_buscarron_path_prefix }}" +matrix_bot_buscarron_container_labels_traefik_rule: "Host(`{{ matrix_bot_buscarron_container_labels_traefik_hostname }}`){% if matrix_bot_buscarron_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_bot_buscarron_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_bot_buscarron_container_labels_traefik_priority: 0 +matrix_bot_buscarron_container_labels_traefik_entrypoints: web-secure +matrix_bot_buscarron_container_labels_traefik_tls: "{{ matrix_bot_buscarron_container_labels_traefik_entrypoints != 'web' }}" +matrix_bot_buscarron_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_bot_buscarron_container_labels_traefik_additional_response_headers_custom` +matrix_bot_buscarron_container_labels_traefik_additional_response_headers: "{{ matrix_bot_buscarron_container_labels_traefik_additional_response_headers_auto | combine(matrix_bot_buscarron_container_labels_traefik_additional_response_headers_custom) }}" +matrix_bot_buscarron_container_labels_traefik_additional_response_headers_auto: {} +matrix_bot_buscarron_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_bot_buscarron_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_bot_buscarron_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_bot_buscarron_container_labels_additional_labels: '' + # A list of extra arguments to pass to the container matrix_bot_buscarron_container_extra_arguments: [] diff --git a/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml b/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml index 94fe93f6..1c2c62e1 100644 --- a/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml +++ b/roles/custom/matrix-bot-buscarron/tasks/setup_install.yml @@ -40,13 +40,16 @@ - {path: "{{ matrix_bot_buscarron_docker_src_files_path }}", when: true} when: "item.when | bool" -- name: Ensure buscarron environment variables file created +- name: Ensure buscarron support files installed ansible.builtin.template: - src: "{{ role_path }}/templates/env.j2" - dest: "{{ matrix_bot_buscarron_config_path }}/env" + src: "{{ role_path }}/templates/{{ item }}.j2" + dest: "{{ matrix_bot_buscarron_config_path }}/{{ item }}" owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" mode: 0640 + with_items: + - env + - labels - name: Ensure buscarron image is pulled community.docker.docker_image: diff --git a/roles/custom/matrix-bot-buscarron/tasks/validate_config.yml b/roles/custom/matrix-bot-buscarron/tasks/validate_config.yml index 3547fc03..51102b73 100644 --- a/roles/custom/matrix-bot-buscarron/tasks/validate_config.yml +++ b/roles/custom/matrix-bot-buscarron/tasks/validate_config.yml @@ -7,4 +7,5 @@ when: "item.when | bool and vars[item.name] == ''" with_items: - {'name': 'matrix_bot_buscarron_password', when: true} + - {'name': 'matrix_bot_buscarron_hostname', when: true} - {'name': 'matrix_bot_buscarron_database_hostname', when: "{{ matrix_bot_buscarron_database_engine == 'postgres' }}"} diff --git a/roles/custom/matrix-bot-buscarron/templates/labels.j2 b/roles/custom/matrix-bot-buscarron/templates/labels.j2 new file mode 100644 index 00000000..5a53805a --- /dev/null +++ b/roles/custom/matrix-bot-buscarron/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_bot_buscarron_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_bot_buscarron_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_bot_buscarron_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_bot_buscarron_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-bot-buscarron-slashless-redirect.redirectregex.regex=({{ matrix_bot_buscarron_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-bot-buscarron-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-bot-buscarron-slashless-redirect'] %} +{% endif %} + +{% if matrix_bot_buscarron_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-bot-buscarron-strip-prefix.stripprefix.prefixes={{ matrix_bot_buscarron_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-bot-buscarron-strip-prefix'] %} +{% endif %} + +{% if matrix_bot_buscarron_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_bot_buscarron_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-bot-buscarron-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-bot-buscarron-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-bot-buscarron.rule={{ matrix_bot_buscarron_container_labels_traefik_rule }} +{% if matrix_bot_buscarron_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-bot-buscarron.priority={{ matrix_bot_buscarron_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-bot-buscarron.service=matrix-bot-buscarron +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-bot-buscarron.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-bot-buscarron.entrypoints={{ matrix_bot_buscarron_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-bot-buscarron.tls={{ matrix_bot_buscarron_container_labels_traefik_tls | to_json }} +{% if matrix_bot_buscarron_container_labels_traefik_tls %} +traefik.http.routers.matrix-bot-buscarron.tls.certResolver={{ matrix_bot_buscarron_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-bot-buscarron.loadbalancer.server.port=8080 +{% endif %} + +{{ matrix_bot_buscarron_container_labels_additional_labels }} diff --git a/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 b/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 index b3ed673d..49c69948 100644 --- a/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 +++ b/roles/custom/matrix-bot-buscarron/templates/systemd/matrix-bot-buscarron.service.j2 @@ -25,6 +25,7 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --read-only \ --network={{ matrix_bot_buscarron_container_network }} \ --env-file={{ matrix_bot_buscarron_config_path }}/env \ + --label-file={{ matrix_bot_buscarron_config_path }}/labels \ --mount type=bind,src={{ matrix_bot_buscarron_data_path }},dst=/data \ {% for arg in matrix_bot_buscarron_container_extra_arguments %} {{ arg }} \ diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 061d8f93..158a1b2b 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -61,11 +61,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule: "Host(`{{ matrix_ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" -matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_hostname: "{{ matrix_server_fqn_buscarron }}" -matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname: "{{ matrix_server_fqn_dimension }}" matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 1e60ba6d..d177d2fb 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -38,17 +38,6 @@ traefik.http.routers.matrix-nginx-proxy-matrix-federation.tls.certResolver={{ ma traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints }} {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_enabled %} -# Buscarron -traefik.http.routers.matrix-nginx-proxy-buscarron.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_rule }} -traefik.http.routers.matrix-nginx-proxy-buscarron.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-buscarron.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_buscarron_tls %} -traefik.http.routers.matrix-nginx-proxy-buscarron.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-buscarron.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - {% if matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled %} # Dimension From ccb5b37842478b2cfc8038d4a01abcf24cec1222 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 15:59:28 +0200 Subject: [PATCH 228/382] Fix some typos on the Buscarron docs page --- docs/configuring-playbook-bot-buscarron.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configuring-playbook-bot-buscarron.md b/docs/configuring-playbook-bot-buscarron.md index dd577854..b21dceef 100644 --- a/docs/configuring-playbook-bot-buscarron.md +++ b/docs/configuring-playbook-bot-buscarron.md @@ -7,13 +7,13 @@ Buscarron is bot that receives HTTP POST submissions of web forms and forwards t ## Decide on a domain and path -By default, Rageshake is configured to use its own dedicated domain (`buscarron.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). +By default, Buscarron is configured to use its own dedicated domain (`buscarron.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). You can override the domain and path like this: ```yaml # Switch to the domain used for Matrix services (`matrix.DOMAIN`), -# so we won't need to add additional DNS records for Rageshake. +# so we won't need to add additional DNS records for Buscarron. matrix_bot_buscarron_hostname: "{{ matrix_server_fqn_matrix }}" # Expose under the /buscarron subpath From 8df7ad47cd1c95a667e9f56e7739de51aca9cadb Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 19:04:28 +0200 Subject: [PATCH 229/382] Fix rageshake project description in defaults/main.yml --- roles/custom/matrix-rageshake/defaults/main.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 456838ed..2dca91b1 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -1,6 +1,7 @@ --- -# rageshake is a issue submission service from matrix.org -# Project source code URL: https://github.com/matrix-org/rageshake/ + +# rageshake is a bug report collection service +# Project source code URL: https://github.com/matrix-org/rageshake matrix_rageshake_enabled: true From 348dd8e76bcd1da24934592ccac9d0f1c2fc2ea9 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Feb 2023 19:36:57 +0200 Subject: [PATCH 230/382] Remove double space --- group_vars/matrix_servers | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index afc1511c..7f378380 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1697,7 +1697,7 @@ matrix_bot_buscarron_container_additional_networks: | ( ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + - ([matrix_nginx_proxy_container_network] if matrix_nginx_proxy_enabled and matrix_bot_buscarron_container_network != matrix_nginx_proxy_container_network else []) + ([matrix_nginx_proxy_container_network] if matrix_nginx_proxy_enabled and matrix_bot_buscarron_container_network != matrix_nginx_proxy_container_network else []) ) | unique }} From 252e542bc552d9e4de48e61b0ef309e5eaaa0f18 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 26 Feb 2023 08:19:51 +0000 Subject: [PATCH 231/382] Update ntfy 2.0.1 -> 2.1.0 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index cbdd37ca..4ebe8a06 100644 --- a/requirements.yml +++ b/requirements.yml @@ -43,7 +43,7 @@ version: v9.3.6-2 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git - version: v2.0.1-0 + version: v2.1.0-0 - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.7-0 From d20ff688dbc31d40f5a4616ee84bafce5a25d1da Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 11:03:42 +0200 Subject: [PATCH 232/382] Add native Traefik support to Sygnal --- docs/configuring-playbook-sygnal.md | 31 ++++++++-- group_vars/matrix_servers | 13 +++- .../matrix-nginx-proxy/defaults/main.yml | 5 -- .../matrix-nginx-proxy/templates/labels.j2 | 11 ---- roles/custom/matrix-sygnal/defaults/main.yml | 59 +++++++++++++++++-- .../tasks/{setup_install.yml => install.yml} | 35 +++++++---- roles/custom/matrix-sygnal/tasks/main.yml | 4 +- .../{setup_uninstall.yml => uninstall.yml} | 0 .../matrix-sygnal/tasks/validate_config.yml | 32 +++++++++- .../custom/matrix-sygnal/templates/labels.j2 | 45 ++++++++++++++ .../systemd/matrix-sygnal.service.j2 | 14 ++++- 11 files changed, 206 insertions(+), 43 deletions(-) rename roles/custom/matrix-sygnal/tasks/{setup_install.yml => install.yml} (78%) rename roles/custom/matrix-sygnal/tasks/{setup_uninstall.yml => uninstall.yml} (100%) create mode 100644 roles/custom/matrix-sygnal/templates/labels.j2 diff --git a/docs/configuring-playbook-sygnal.md b/docs/configuring-playbook-sygnal.md index 1a344530..786498c6 100644 --- a/docs/configuring-playbook-sygnal.md +++ b/docs/configuring-playbook-sygnal.md @@ -11,6 +11,31 @@ See the project's [documentation](https://github.com/matrix-org/sygnal) to learn This optional playbook component is only useful to people who develop/build their own Matrix client applications themselves. +## Decide on a domain and path + +By default, Sygnal is configured to use its own dedicated domain (`sygnal.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). + +You can override the domain and path like this: + +```yaml +# Switch to the domain used for Matrix services (`matrix.DOMAIN`), +# so we won't need to add additional DNS records for Sygnal. +matrix_sygnal_hostname: "{{ matrix_server_fqn_matrix }}" + +# Expose under the /sygnal subpath +matrix_sygnal_path_prefix: /sygnal +``` + +**NOTE**: When using `matrix-nginx-proxy` instead of Traefik, you won't be able to override the path prefix. You can only override the domain, but that needs to happen using another variable: `matrix_server_fqn_sygnal` (e.g. `matrix_server_fqn_sygnal: "push.{{ matrix_domain }}"`). + + +## Adjusting DNS records + +Once you've decided on the domain and path, **you may need to adjust your DNS** records to point the Sygnal domain to the Matrix server. + +If you've decided to reuse the `matrix.` domain, you won't need to do any extra DNS configuration. + + ## Adjusting the playbook configuration Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file (adapt to your needs): @@ -55,9 +80,7 @@ To do that, the above example configuration: ## Installing -Don't forget to add `sygnal.` to DNS as described in [Configuring DNS](configuring-dns.md) before running the playbook. - -After configuring the playbook, run the [installation](installing.md) command again: +After configuring the playbook and potentially [adjusting your DNS records](#adjusting-dns-records), run the [installation](installing.md) command: ``` ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start @@ -66,6 +89,6 @@ ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start ## Usage -To make use of your Sygnal installation, you'd need to build your own Matrix client application, which uses the same API keys (for [GCM/FCM](https://firebase.google.com/docs/cloud-messaging/)) and certificates (for [APNS](https://developer.apple.com/notifications/)) and is also pointed to `https://sygnal.DOMAIN` as the configured push server. +To make use of your Sygnal installation, you'd need to build your own Matrix client application, which uses the same API keys (for [GCM/FCM](https://firebase.google.com/docs/cloud-messaging/)) and certificates (for [APNS](https://developer.apple.com/notifications/)) and is to your Sygnal URL endpoint (e.g. `https://sygnal.DOMAIN`). Refer to Sygnal's [Notes for application developers](https://github.com/matrix-org/sygnal/blob/master/docs/applications.md) document. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 7f378380..359c6e3f 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2443,7 +2443,7 @@ matrix_nginx_proxy_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" matrix_nginx_proxy_proxy_jitsi_manage_wellknown: "{{ matrix_jitsi_require_well_known }}" matrix_nginx_proxy_proxy_grafana_enabled: "{{ grafana_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" -matrix_nginx_proxy_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" +matrix_nginx_proxy_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_ntfy_enabled: "{{ ntfy_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" @@ -2455,7 +2455,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" -matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled: "{{ matrix_sygnal_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: "{{ matrix_corporal_enabled and matrix_corporal_http_api_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corporal:41081" @@ -2931,8 +2930,18 @@ matrix_sygnal_enabled: false # If someone instals Prometheus via the playbook, they most likely wish to monitor Sygnal. matrix_sygnal_metrics_prometheus_enabled: "{{ matrix_prometheus_enabled }}" +matrix_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}" + +matrix_sygnal_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-sygnal' }}" + +matrix_sygnal_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}" + matrix_sygnal_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '6000') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_sygnal_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +matrix_sygnal_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_sygnal_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" + ###################################################################### # # /matrix-sygnal diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 158a1b2b..a098a098 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -81,11 +81,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_hostname: "{{ matrix_ser matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_hostname }}`)" -matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}" -matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_hostname }}`)" - # matrix_nginx_proxy_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. # See `../templates/labels.j2` for details. # diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index d177d2fb..6afa6029 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -86,17 +86,6 @@ traefik.http.routers.matrix-nginx-proxy-jitsi.tls.certResolver={{ matrix_nginx_p traefik.http.routers.matrix-nginx-proxy-jitsi.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_enabled %} -# Sygnal -traefik.http.routers.matrix-nginx-proxy-sygnal.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_rule }} -traefik.http.routers.matrix-nginx-proxy-sygnal.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-sygnal.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_sygnal_tls %} -traefik.http.routers.matrix-nginx-proxy-sygnal.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-sygnal.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - traefik.http.services.matrix-nginx-proxy-web.loadbalancer.server.port=8080 diff --git a/roles/custom/matrix-sygnal/defaults/main.yml b/roles/custom/matrix-sygnal/defaults/main.yml index 1268d602..f3c1df4e 100644 --- a/roles/custom/matrix-sygnal/defaults/main.yml +++ b/roles/custom/matrix-sygnal/defaults/main.yml @@ -1,28 +1,77 @@ --- + # Sygnal is a reference Push Gateway for Matrix. # To make use of it for delivering push notificatins, you'll need to develop/build your own Matrix app. # Project source code URL: https://github.com/matrix-org/sygnal -matrix_sygnal_enabled: false +matrix_sygnal_enabled: true + +# The hostname at which Sygnal is served. +matrix_sygnal_hostname: '' + +# The path at which Sygnal is exposed. +# This value must either be `/` or not end with a slash (e.g. `/sygnal`). +matrix_sygnal_path_prefix: / + +matrix_sygnal_version: v0.12.0 matrix_sygnal_base_path: "{{ matrix_base_data_path }}/sygnal" matrix_sygnal_config_path: "{{ matrix_sygnal_base_path }}/config" matrix_sygnal_data_path: "{{ matrix_sygnal_base_path }}/data" -matrix_sygnal_version: v0.12.0 -matrix_sygnal_docker_image: "{{ matrix_container_global_registry_prefix }}matrixdotorg/sygnal:{{ matrix_sygnal_version }}" -matrix_sygnal_docker_image_force_pull: "{{ matrix_sygnal_docker_image.endswith(':latest') }}" - # List of systemd services that matrix-sygnal.service depends on. matrix_sygnal_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-sygnal.service wants matrix_sygnal_systemd_wanted_services_list: [] +matrix_sygnal_docker_image: "{{ matrix_sygnal_docker_image_registry_prefix }}matrixdotorg/sygnal:{{ matrix_sygnal_docker_image_tag }}" +matrix_sygnal_docker_image_tag: "{{ matrix_sygnal_version }}" +matrix_sygnal_docker_image_registry_prefix: "{{ matrix_container_global_registry_prefix }}" +matrix_sygnal_docker_image_force_pull: "{{ matrix_sygnal_docker_image.endswith(':latest') }}" + +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_sygnal_container_network: "{{ matrix_sygnal_identifier }}" + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_sygnal_container_additional_networks: [] + # Controls whether the matrix-sygnal container exposes its HTTP port (tcp/6000 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:6000"), or empty string to not expose. matrix_sygnal_container_http_host_bind_port: '' +# matrix_sygnal_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_sygnal_container_labels_additional_labels`. +matrix_sygnal_container_labels_traefik_enabled: true +matrix_sygnal_container_labels_traefik_docker_network: "{{ matrix_sygnal_container_network }}" +matrix_sygnal_container_labels_traefik_hostname: "{{ matrix_sygnal_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/sygnal`). +matrix_sygnal_container_labels_traefik_path_prefix: "{{ matrix_sygnal_path_prefix }}" +matrix_sygnal_container_labels_traefik_rule: "Host(`{{ matrix_sygnal_container_labels_traefik_hostname }}`){% if matrix_sygnal_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_sygnal_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_sygnal_container_labels_traefik_priority: 0 +matrix_sygnal_container_labels_traefik_entrypoints: web-secure +matrix_sygnal_container_labels_traefik_tls: "{{ matrix_sygnal_container_labels_traefik_entrypoints != 'web' }}" +matrix_sygnal_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_sygnal_container_labels_traefik_additional_response_headers_custom` +matrix_sygnal_container_labels_traefik_additional_response_headers: "{{ matrix_sygnal_container_labels_traefik_additional_response_headers_auto | combine(matrix_sygnal_container_labels_traefik_additional_response_headers_custom) }}" +matrix_sygnal_container_labels_traefik_additional_response_headers_auto: {} +matrix_sygnal_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_sygnal_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_sygnal_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_sygnal_container_labels_additional_labels: '' + # A list of extra arguments to pass to the container matrix_sygnal_container_extra_arguments: [] diff --git a/roles/custom/matrix-sygnal/tasks/setup_install.yml b/roles/custom/matrix-sygnal/tasks/install.yml similarity index 78% rename from roles/custom/matrix-sygnal/tasks/setup_install.yml rename to roles/custom/matrix-sygnal/tasks/install.yml index 7c9caf38..7ee75a94 100644 --- a/roles/custom/matrix-sygnal/tasks/setup_install.yml +++ b/roles/custom/matrix-sygnal/tasks/install.yml @@ -1,16 +1,5 @@ --- -- name: Ensure Sygnal image is pulled - community.docker.docker_image: - name: "{{ matrix_sygnal_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_sygnal_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_sygnal_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - - name: Ensure Sygnal paths exists ansible.builtin.file: path: "{{ item }}" @@ -31,6 +20,30 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" +- name: Ensure Sygnal labels installed + ansible.builtin.template: + src: "{{ role_path }}/templates/labels.j2" + dest: "{{ matrix_sygnal_base_path }}/labels" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure Sygnal image is pulled + community.docker.docker_image: + name: "{{ matrix_sygnal_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_sygnal_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_sygnal_docker_image_force_pull }}" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure Sygnal container network is created + community.general.docker_network: + name: "{{ matrix_sygnal_container_network }}" + driver: bridge + - name: Ensure matrix-sygnal.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-sygnal.service.j2" diff --git a/roles/custom/matrix-sygnal/tasks/main.yml b/roles/custom/matrix-sygnal/tasks/main.yml index ed801526..ba4cb876 100644 --- a/roles/custom/matrix-sygnal/tasks/main.yml +++ b/roles/custom/matrix-sygnal/tasks/main.yml @@ -5,7 +5,7 @@ ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_sygnal_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" tags: - setup-all - setup-sygnal @@ -14,7 +14,7 @@ - block: - when: not matrix_sygnal_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" tags: - setup-all - setup-sygnal diff --git a/roles/custom/matrix-sygnal/tasks/setup_uninstall.yml b/roles/custom/matrix-sygnal/tasks/uninstall.yml similarity index 100% rename from roles/custom/matrix-sygnal/tasks/setup_uninstall.yml rename to roles/custom/matrix-sygnal/tasks/uninstall.yml diff --git a/roles/custom/matrix-sygnal/tasks/validate_config.yml b/roles/custom/matrix-sygnal/tasks/validate_config.yml index 277bd1b2..1cdf2038 100644 --- a/roles/custom/matrix-sygnal/tasks/validate_config.yml +++ b/roles/custom/matrix-sygnal/tasks/validate_config.yml @@ -1,7 +1,37 @@ --- +- name: Fail if required Sygnal settings not defined + ansible.builtin.fail: + msg: > + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_sygnal_hostname + - matrix_sygnal_path_prefix + - matrix_sygnal_container_network + +- when: matrix_sygnal_container_labels_traefik_enabled | bool + block: + - name: Fail if required Sygnal Traefik settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_sygnal_container_labels_traefik_hostname + - matrix_sygnal_container_labels_traefik_path_prefix + + # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). + # Knowing that `matrix_sygnal_container_labels_traefik_path_prefix` does not end with a slash + # ensures we know how to set these routes up without having to do "does it end with a slash" checks elsewhere. + - name: Fail if matrix_sygnal_container_labels_traefik_path_prefix ends with a slash + ansible.builtin.fail: + msg: >- + matrix_sygnal_container_labels_traefik_path_prefix (`{{ matrix_sygnal_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/sygnal`). + when: "matrix_sygnal_container_labels_traefik_path_prefix != '/' and matrix_sygnal_container_labels_traefik_path_prefix[-1] == '/'" + - name: Fail if no Sygnal apps defined ansible.builtin.fail: msg: >- Enabling Sygnal requires that you specify at least one app in `matrix_sygnal_apps` - when: "matrix_sygnal_enabled and matrix_sygnal_apps | length == 0" + when: "matrix_sygnal_apps | length == 0" diff --git a/roles/custom/matrix-sygnal/templates/labels.j2 b/roles/custom/matrix-sygnal/templates/labels.j2 new file mode 100644 index 00000000..96480589 --- /dev/null +++ b/roles/custom/matrix-sygnal/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_sygnal_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_sygnal_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_sygnal_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_sygnal_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-sygnal-slashless-redirect.redirectregex.regex=({{ matrix_sygnal_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-sygnal-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-sygnal-slashless-redirect'] %} +{% endif %} + +{% if matrix_sygnal_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-sygnal-strip-prefix.stripprefix.prefixes={{ matrix_sygnal_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-sygnal-strip-prefix'] %} +{% endif %} + +{% if matrix_sygnal_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_sygnal_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-sygnal-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-sygnal-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-sygnal.rule={{ matrix_sygnal_container_labels_traefik_rule }} +{% if matrix_sygnal_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-sygnal.priority={{ matrix_sygnal_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-sygnal.service=matrix-sygnal +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-sygnal.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-sygnal.entrypoints={{ matrix_sygnal_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-sygnal.tls={{ matrix_sygnal_container_labels_traefik_tls | to_json }} +{% if matrix_sygnal_container_labels_traefik_tls %} +traefik.http.routers.matrix-sygnal.tls.certResolver={{ matrix_sygnal_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-sygnal.loadbalancer.server.port=6000 +{% endif %} + +{{ matrix_sygnal_container_labels_additional_labels }} diff --git a/roles/custom/matrix-sygnal/templates/systemd/matrix-sygnal.service.j2 b/roles/custom/matrix-sygnal/templates/systemd/matrix-sygnal.service.j2 index 646314df..49b513cd 100644 --- a/roles/custom/matrix-sygnal/templates/systemd/matrix-sygnal.service.j2 +++ b/roles/custom/matrix-sygnal/templates/systemd/matrix-sygnal.service.j2 @@ -16,15 +16,18 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-sygnal 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-sygnal 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-sygnal \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-sygnal \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --env=SYGNAL_CONF=/config/sygnal.yaml \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_sygnal_container_network }} \ {% if matrix_sygnal_container_http_host_bind_port %} -p {{ matrix_sygnal_container_http_host_bind_port }}:6000 \ {% endif %} + --label-file={{ matrix_sygnal_base_path }}/labels \ --mount type=bind,src={{ matrix_sygnal_config_path }},dst=/config \ --mount type=bind,src={{ matrix_sygnal_data_path }},dst=/data \ {% for arg in matrix_sygnal_container_extra_arguments %} @@ -32,8 +35,15 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endfor %} {{ matrix_sygnal_docker_image }} +{% for network in matrix_sygnal_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-sygnal +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-sygnal + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-sygnal 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-sygnal 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-sygnal From e625e9aa5bd6c8685b77ea6de36c300e4a45e44d Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 26 Feb 2023 10:41:40 +0000 Subject: [PATCH 233/382] fix postgres version detection in the borg role --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 4ebe8a06..56fa7701 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git - version: v1.2.3-1.7.7-0 + version: v1.2.3-1.7.7-1 - src: git+https://gitlab.com/etke.cc/roles/grafana.git version: v9.3.6-2 From 67e97cd2dc46a73acc6c89c5c93c77e17aae932a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 15:54:30 +0200 Subject: [PATCH 234/382] Redo etke.cc section in the README --- README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index d284909f..f57db148 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,15 @@ We run all services in [Docker](https://www.docker.com/) containers (see [the co [Installation](docs/README.md) (upgrades) and some maintenance tasks are automated using [Ansible](https://www.ansible.com/) (see [our Ansible guide](docs/ansible.md)). +## Self-hosting or SaaS + +This Ansible playbook tries to make self-hosting and maintaining a Matrix server fairly easy. Still, running any service smoothly requires knowledge, time and effort. + +If you like the [FOSS](https://en.wikipedia.org/wiki/Free_and_open-source_software) spirit of this Ansible playbook, but prefer to put the responsibility on someone else, you can also [get a managed Matrix server from etke.cc](https://etke.cc/) - a service built on top of this Ansible playbook, which can help you run a Matrix server with ease. + +If you like learning and experimentation, but would rather reduce future maintenance effort, you can even go for a hybrid approach - self-hosting manually using this Ansible playbook at first and then transferring server maintenance to etke.cc at a later time. + + ## Supported services Using this playbook, you can get the following list of services configured on your server. Basically, this playbook aims to get you up-and-running with all the necessities around Matrix, without you having to do anything else. @@ -178,8 +187,3 @@ When updating the playbook, refer to [the changelog](CHANGELOG.md) to catch up w - IRC channel: `#matrix-docker-ansible-deploy` on the [Libera Chat](https://libera.chat/) IRC network (irc.libera.chat:6697) - GitHub issues: [spantaleev/matrix-docker-ansible-deploy/issues](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues) - - -## Services by the community - -- [etke.cc](https://etke.cc) - matrix-docker-ansible-deploy and system stuff "as a service". That service will create your matrix homeserver on your domain and server (doesn't matter if it's cloud provider or on an old laptop in the corner of your room), (optional) maintains it (server's system updates, cleanup, security adjustments, tuning, etc.; matrix homeserver updates & maintenance) and (optional) provide full-featured email service for your domain From 71ae825f6aa70a61fb457a213f3bc69787c1d221 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 16:14:05 +0200 Subject: [PATCH 235/382] Switch default reverse-proxy to Traefik, force reverse-proxy choice on existing users --- CHANGELOG.md | 62 +++++++++++ README.md | 5 +- docs/configuring-playbook-nginx.md | 3 +- docs/configuring-playbook-own-webserver.md | 54 ++++----- docs/configuring-playbook-ssl-certificates.md | 104 +++--------------- docs/configuring-playbook-traefik.md | 48 ++++++++ docs/configuring-playbook.md | 6 +- examples/vars.yml | 7 +- roles/custom/matrix-base/defaults/main.yml | 2 +- .../matrix-base/tasks/validate_config.yml | 8 ++ 10 files changed, 175 insertions(+), 124 deletions(-) create mode 100644 docs/configuring-playbook-traefik.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a49de417..5cf821f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,65 @@ +# 2023-02-26 + +## Traefik is the default reverse-proxy now + +**TLDR**: new installations will now default to Traefik as their reverse-proxy. Existing users need to explicitly choose their reverse-proxy type. [Switching to Traefik](#how-do-i-switch-my-existing-setup-to-traefik) is strongly encouraged. `matrix-nginx-proxy` may break over time and will ultimately be removed. + +As mentioned 2 weeks ago in [(Backward Compatibility) Reverse-proxy configuration changes and initial Traefik support](#backward-compatibility-reverse-proxy-configuration-changes-and-initial-traefik-support), the playbook is moving to Traefik as its default SSL-terminating reverse-proxy. + +Until now, we've been doing the migration gradually and keeping full backward compatibility. New installations were defaulting to `matrix-nginx-proxy` (just like before), while existing installations were allowed to remain on `matrix-nginx-proxy` as well. This makes things very difficult for us, because we need to maintain and think about lots of different setups: + +- Traefik managed by the playbook +- Traefik managed by the user in another way +- another reverse-proxy on the same host (`127.0.0.1` port exposure) +- another reverse-proxy on another host (`0.0.0.0` port exposure) +- `matrix-nginx-proxy` - an `nginx` container managed by the playbook +- `nginx` webserver operated by the user, running without a container on the same server + +Each change we do and each new feature that comes in needs to support all these different ways of reverse-proxying. Because `matrix-nginx-proxy` was the default and pretty much everyone was (and still is) using it, means that new PRs also come with `matrix-nginx-proxy` as their main focus and Traefik as an afterthought, which means we need to spend hours fixing up Traefik support. + +We can't spend all this time maintaining so many different configurations anymore. Traefik support has been an option for 2 weeks and lots of people have already migrated their server and have tested things out. Traefik is what we use and preferentially test for. + +It's time for the **next step in our migration process** to Traefik and elimination of `matrix-nginx-proxy`: + +- Traefik is now the default reverse-proxy for new installations +- All existing users need to explicitly choose their reverse-proxy type by defining the `matrix_playbook_reverse_proxy_type` variable in their `vars.yml` configuration file. We strongly encourage existing users to [switch the Traefik](#how-to-switch-an-existing-setup-to-traefik), as the nginx setup is bound to become more and more broken over time until it's ultimately removed + +### How do I switch my existing setup to Traefik? + +**For users who are on `matrix-nginx-proxy`** (the default reverse-proxy provided by the playbook), switching to Traefik can happen with a simple configuration change. Follow this section from 2 weeks ago: [How do I explicitly switch to Traefik right now?](#how-do-i-explicitly-switch-to-traefik-right-now). + +If you experience trouble: + +1. Follow [How do I remain on matrix-nginx-proxy?](#how-do-i-remain-on-matrix-nginx-proxy) to bring your server back online using the old reverse-proxy +2. Ask for help in our [support channels](README.md#support) +3. Try switching to Traefik again later + +**For users with a more special reverse-proxying setup** (another nginx server, Apache, Caddy, etc.), the migration may not be so smooth. Follow the [Using your own webserver](docs/configuring-playbook-own-webserver.md) guide. Ideally, your custom reverse-proxy will be configured in such a way that it **fronts the Traefik reverse-proxy** provided by the playbook. Other means of reverse-proxying are more fragile and may be deprecated in the future. + +### I already use my own Traefik server. How do I plug that in? + +See the [Traefik managed by the playbook](docs/configuring-playbook-own-webserver.md#traefik-managed-by-the-playbook) section. + +### Why is matrix-nginx-proxy used even after switching to Traefik? + +This playbook manages many different services. All these services were initially integrated with `matrix-nginx-proxy`. + +While we migrate all these components to have native Traefik support, some still go through nginx internally (Traefik -> local `matrix-nginx-proxy` -> component). +As time goes on, internal reliance on `matrix-nginx-proxy` will gradually decrease until it's completely removed. + +### How do I remain on matrix-nginx-proxy? + +Most new work and testing targets Traefik, so remaining on nginx is **not** "the good old stable" option, but rather the "still available, but largely untested and likely to be broken very soon" option. + +To proceed regardless of this warning, add `matrix_playbook_reverse_proxy_type: playbook-managed-nginx` to your configuration. + +At some point in the **near** future (days, or even weeks at most), we hope to completely get rid of `matrix-nginx-proxy` (or break it enough to make it unusable), so you **will soon be forced to migrate** anyway. Plan your migration accordingly. + +### How do I keep using my own other reverse-proxy? + +We recommend that you follow the guide for [Fronting the integraed reverse-proxy webserver with another reverse-proxy](docs/configuring-playbook-own-webserver.md#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy). + + # 2023-02-25 ## Rageshake support diff --git a/README.md b/README.md index f57db148..dd446725 100644 --- a/README.md +++ b/README.md @@ -61,8 +61,9 @@ Services that run on the server to make the various parts of your installation w | ---- | -------- | ----------- | ------------- | | [PostgreSQL](https://www.postgresql.org/)| ✓ | Database for Synapse. [Using an external PostgreSQL server](docs/configuring-playbook-external-postgres.md) is also possible. | [Link](docs/configuring-playbook-external-postgres.md) | | [Coturn](https://github.com/coturn/coturn) | ✓ | STUN/TURN server for WebRTC audio/video calls | [Link](docs/configuring-playbook-turn.md) | -| [nginx](http://nginx.org/) | ✓ | Web server, listening on ports 80 and 443 - standing in front of all the other services. Using your own webserver [is possible](docs/configuring-playbook-own-webserver.md) | [Link](docs/configuring-playbook-nginx.md) | -| [Let's Encrypt](https://letsencrypt.org/) | ✓ | Free SSL certificate, which secures the connection to the Synapse server and the Element web UI | [Link](docs/configuring-playbook-ssl-certificates.md) | +| [Traefik](https://doc.traefik.io/traefik/) | ✓ | Web server, listening on ports 80, 443 and 8448 - standing in front of all the other services. Using your own webserver [is possible](docs/configuring-playbook-own-webserver.md) | [Link](docs/configuring-playbook-traefik.md) | +| [nginx](http://nginx.org/) | x | (Deprecated) Web server, listening on ports 80, 443 and 8448 - standing in front of all the other services. Deprecated in favor of Traefik | [Link](docs/configuring-playbook-nginx.md) | +| [Let's Encrypt](https://letsencrypt.org/) | ✓ | Free SSL certificate, which secures the connection to all components | [Link](docs/configuring-playbook-ssl-certificates.md) | | [ma1sd](https://github.com/ma1uta/ma1sd) | x | Matrix Identity Server | [Link](docs/configuring-playbook-ma1sd.md) | [Exim](https://www.exim.org/) | ✓ | Mail server, through which all Matrix services send outgoing email (can be configured to relay through another SMTP server) | [Link](docs/configuring-playbook-email.md) | | [Dimension](https://github.com/turt2live/matrix-dimension) | x | An open source integrations manager for matrix clients | [Link](docs/configuring-playbook-dimension.md) | diff --git a/docs/configuring-playbook-nginx.md b/docs/configuring-playbook-nginx.md index 96f854d0..43cff4a8 100644 --- a/docs/configuring-playbook-nginx.md +++ b/docs/configuring-playbook-nginx.md @@ -1,7 +1,6 @@ # Configure Nginx (optional, advanced) -By default, this playbook installs its own nginx webserver (in a Docker container) which listens on ports 80 and 443. -If that's alright, you can skip this. +**Note**: the playbook is [in the process of moving to Traefik](../CHANGELOG.md#reverse-proxy-configuration-changes-and-initial-traefik-support). Traefik is already the default reverse-proxy for new installations and existing users are also strongly encouraged to switch to Traefik. As such, this **nginx documentation below may be incomplete or misleading**. ## Using Nginx status diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 22170f40..3b9d822d 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -1,4 +1,4 @@ -# Using your own webserver, instead of this playbook's nginx proxy (optional, advanced) +# Using your own webserver, instead of this playbook's Traefik reverse-proxy (optional, advanced) **Note**: the playbook is [in the process of moving to Traefik](../CHANGELOG.md#reverse-proxy-configuration-changes-and-initial-traefik-support). The **documentation below may be incomplete or misleading**. @@ -117,17 +117,36 @@ There are 2 ways to go about it: This method is about leaving the integrated reverse-proxy webserver be, but making it not get in the way (using up important ports, trying to retrieve SSL certificates, etc.). -If you wish to use another webserver, the integrated nginx reverse-proxy webserver usually gets in the way because it attempts to fetch SSL certificates and binds to ports 80, 443 and 8448 (if Matrix Federation is enabled). +If you wish to use another webserver, the integrated reverse-proxy webserver usually gets in the way because it attempts to fetch SSL certificates and binds to ports 80, 443 and 8448 (if Matrix Federation is enabled). You can disable such behavior and make the integrated reverse-proxy webserver only serve traffic locally (or over a local network). This is the recommended way for using another reverse-proxy, because the integrated one would act as a black box and wire all Matrix services correctly. You would only need to reverse-proxy a few individual domains and ports over to it. -**For `matrix-nginx-proxy`** fronted by another reverse-proxy, you would need some configuration like this: +**For Traefik** fronted by another reverse-proxy, you would need some configuration like this: + +```yaml +matrix_playbook_reverse_proxy_type: playbook-managed-traefik + +# Ensure that public urls use https +matrix_playbook_ssl_enabled: true + +# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval +devture_traefik_config_entrypoint_web_secure_enabled: false + +devture_traefik_container_web_host_bind_port: '127.0.0.1:81' + +devture_traefik_additional_entrypoints_auto: + - name: matrix-federation + port: "{{ matrix_federation_public_port }}" + host_bind_port: "127.0.0.1:{{ matrix_federation_public_port }}" + config: {} +``` + +(Deprecated) **For `matrix-nginx-proxy`** fronted by another reverse-proxy, you would need some configuration like this: ```yaml -# playbook-managed-proxy is the default right now, so we can keep this commented out. -# matrix_playbook_reverse_proxy_type: playbook-managed-nginx +matrix_playbook_reverse_proxy_type: playbook-managed-nginx # Ensure that public urls use https matrix_playbook_ssl_enabled: true @@ -150,28 +169,9 @@ matrix_nginx_proxy_container_http_host_bind_port: '127.0.0.1:81' matrix_nginx_proxy_container_federation_host_bind_port: '127.0.0.1:8449' ``` -**For Traefik** fronted by another reverse-proxy, you would need some configuration like this: - -```yaml -matrix_playbook_reverse_proxy_type: playbook-managed-traefik - -# Ensure that public urls use https -matrix_playbook_ssl_enabled: true - -# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval -devture_traefik_config_entrypoint_web_secure_enabled: false - -devture_traefik_container_web_host_bind_port: '127.0.0.1:81' - -devture_traefik_additional_entrypoints_auto: - - name: matrix-federation - port: "{{ matrix_federation_public_port }}" - host_bind_port: "127.0.0.1:{{ matrix_federation_public_port }}" - config: {} -``` - If you'll be fronting with a reverse-proxy that lives on another machine (not on the same one as Matrix), you need to replace `127.0.0.1` in the above configurations with `0.0.0.0` or another network interface. + ### Using no reverse-proxy on the Matrix side at all Instead of [Fronting the integrated reverse-proxy webserver with another reverse-proxy](#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy), you can also go another way -- completely disabling the playbook-managed reverse-proxy. You would then need to reverse-proxy from your own webserver directly to Matrix services. @@ -182,6 +182,8 @@ If your webserver is on the same machine, sure your web server user (something l #### Using your own nginx reverse-proxy running on the same machine +**WARNING**: this type of setup is not maintained and will be removed in the future. We recommend that you go for [Fronting the integrated reverse-proxy webserver with another reverse-proxy](#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy) instead. + If you'll be using `nginx` running on the same machine (not in a container), you can make the playbook help you generate configuration for `nginx` with this configuration: ```yaml @@ -202,6 +204,8 @@ You can most likely directly use the config files installed by this playbook at: #### Using your own reverse-proxy running on the same machine or elsewhere +**WARNING**: this is difficult to set up, likely not very well supported and will be removed in the future. We recommend that you go for [Fronting the integrated reverse-proxy webserver with another reverse-proxy](#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy) instead. + To reverse-proxy manually for each service, use configuration like this: ```yaml diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index 606160da..60d5b135 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -1,112 +1,34 @@ # Adjusting SSL certificate retrieval (optional, advanced) -By default, this playbook retrieves and auto-renews free SSL certificates from [Let's Encrypt](https://letsencrypt.org/) for the domains it needs (`matrix.` and possibly `element.`) +By default, this playbook retrieves and auto-renews free SSL certificates from [Let's Encrypt](https://letsencrypt.org/) for the domains it needs (e.g. `matrix.` and others) -Those certificates are used when configuring the nginx reverse proxy installed by this playbook. -They can also be used for configuring [your own webserver](configuring-playbook-own-webserver.md), in case you're not using the integrated nginx server provided by the playbook. +This guide is about using the integrated Traefik server and doesn't apply if you're using [your own webserver](configuring-playbook-own-webserver.md). -If you need to retrieve certificates for other domains (e.g. your base domain) or more control over certificate retrieval, read below. -Things discussed in this document: +## Using staging Let's Encrypt certificates instead of real ones -- [Using self-signed SSL certificates](#using-self-signed-ssl-certificates), if you can't use Let's Encrypt or just need a test setup +For testing purposes, you may wish to use staging certificates provide by Let's Encrypt. -- [Using your own SSL certificates](#using-your-own-ssl-certificates), if you don't want to or can't use Let's Encrypt certificates, but are still interested in using the integrated nginx reverse proxy server - -- [Not bothering with SSL certificates](#not-bothering-with-ssl-certificates), if you're using [your own webserver](configuring-playbook-own-webserver.md) and would rather this playbook leaves SSL certificate management to you - -- [Obtaining SSL certificates for additional domains](#obtaining-ssl-certificates-for-additional-domains), if you'd like to host additional domains on the Matrix server and would like the playbook to help you obtain and renew certificates for those domains automatically - - -## Using self-signed SSL certificates - -For private deployments (not publicly accessible from the internet), you may not be able to use Let's Encrypt certificates. - -If self-signed certificates are alright with you, you can ask the playbook to generate such for you with the following configuration: - -```yaml -matrix_ssl_retrieval_method: self-signed -``` - -If you get a `Cannot reach homeserver` error in Element, you will have to visit `https://matrix.` in your browser and agree to the certificate exception before you can login. - - -## Using your own SSL certificates - -If you'd like to manage SSL certificates by yourself and have the playbook use your certificate files, you can use the following configuration: - -```yaml -matrix_ssl_retrieval_method: manually-managed -``` - -With such a configuration, the playbook would expect you to drop the SSL certificate files in the directory specified by `matrix_ssl_config_dir_path` (`/matrix/ssl/config` by default) obeying the following hierarchy: - -- `/live//fullchain.pem` -- `/live//privkey.pem` -- `/live//chain.pem` - -where `` refers to the domains that you need (usually `matrix.` and `element.`). - - -## Not bothering with SSL certificates - -If you're [using an external web server](configuring-playbook-own-webserver.md) which is not nginx, or you would otherwise want to manage its certificates without this playbook getting in the way, you can completely disable SSL certificate management with the following configuration: +You can do this with the following configuration: ```yaml -matrix_ssl_retrieval_method: none +devture_traefik_config_certificatesResolvers_acme_use_staging: true ``` -With such a configuration, no certificates will be retrieved at all. You're free to manage them however you want. - -## Obtaining SSL certificates for additional domains +## Disabling SSL termination -The playbook tries to be smart about the certificates it will obtain for you. +For testing or other purposes, you may wish to install services without SSL termination and have services exposed to `http://` instead of `https://`. -By default, it obtains certificates for: -- `matrix.` (`matrix_server_fqn_matrix`) -- possibly for `element.`, unless you have disabled the [Element client component](configuring-playbook-client-element.md) using `matrix_client_element_enabled: false` -- possibly for `riot.`, if you have explicitly enabled Riot to Element redirection (for background compatibility) using `matrix_nginx_proxy_proxy_riot_compat_redirect_enabled: true` -- possibly for `hydrogen.`, if you have explicitly [set up Hydrogen client](configuring-playbook-client-hydrogen.md). -- possibly for `cinny.`, if you have explicitly [set up Cinny client](configuring-playbook-client-cinny.md). -- possibly for `dimension.`, if you have explicitly [set up Dimension](configuring-playbook-dimension.md). -- possibly for `goneb.`, if you have explicitly [set up Go-NEB bot](configuring-playbook-bot-go-neb.md). -- possibly for `jitsi.`, if you have explicitly [set up Jitsi](configuring-playbook-jitsi.md). -- possibly for `stats.`, if you have explicitly [set up Grafana](configuring-playbook-prometheus-grafana.md). -- possibly for `sygnal.`, if you have explicitly [set up Sygnal](configuring-playbook-sygnal.md). -- possibly for `ntfy.`, if you have explicitly [set up ntfy](configuring-playbook-ntfy.md). -- possibly for your base domain (``), if you have explicitly configured [Serving the base domain](configuring-playbook-base-domain-serving.md) - -If you are hosting other domains on the Matrix machine, you can make the playbook obtain and renew certificates for those other domains too. -To do that, simply define your own custom configuration like this: +You can do this with the following configuration: ```yaml -# In this example, we retrieve 2 extra certificates, -# one for the base domain (in the `matrix_domain` variable) and one for a hardcoded domain. -# Adding any other additional domains (hosted on the same machine) is possible. -matrix_ssl_additional_domains_to_obtain_certificates_for: - - '{{ matrix_domain }}' - - 'another.domain.example.com' +devture_traefik_config_entrypoint_web_secure_enabled: false ``` -After redefining `matrix_ssl_domains_to_obtain_certificates_for`, to actually obtain certificates you should: - -- make sure the web server occupying port 80 is stopped. If you are using matrix-nginx-proxy server (which is the default for this playbook), you need to stop it temporarily by running `systemctl stop matrix-nginx-proxy` on the server. -- re-run the SSL part of the playbook and restart all services: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-ssl,start` - -The certificate files would be made available in `/matrix/ssl/config/live//...`. - -For automated certificate renewal to work, each port `80` vhost for each domain you are obtaining certificates for needs to forward requests for `/.well-known/acme-challenge` to the certbot container we use for renewal. - -See how this is configured for the `matrix.` subdomain in `/matrix/nginx-proxy/conf.d/matrix-domain.conf` -Don't be alarmed if the above configuration file says port `8080`, instead of port `80`. It's due to port mapping due to our use of containers. - - -## Specify the SSL private key algorithm +## Using self-signed SSL certificates -If you'd like to [specify the private key type](https://eff-certbot.readthedocs.io/en/stable/using.html#using-ecdsa-keys) used with Let's Encrypt, define your own custom configuration like this: +Using self-signed certificates with Traefik is a somewhat involved processes, where you need to manually mount the files into the container and adjust the "static" configuration to refer to them. -```yaml -matrix_ssl_lets_encrypt_key_type: ecdsa -``` +Feel free to research this approach on your own and improve this guide! diff --git a/docs/configuring-playbook-traefik.md b/docs/configuring-playbook-traefik.md new file mode 100644 index 00000000..4f6a7335 --- /dev/null +++ b/docs/configuring-playbook-traefik.md @@ -0,0 +1,48 @@ +# Configure Traefik (optional, advanced) + +By default, this playbook installs and manages a [Traefik](https://doc.traefik.io/traefik/) reverse-proxy server, powered by the [com.devture.ansible.role.traefik](https://github.com/devture/com.devture.ansible.role.traefik) Ansible role. + +This Ansible role support various configuration options. Feel free to consult its `default/main.yml` variables file. + + +## Adjusting SSL certificate retrieval + +See the dedicated [Adjusting SSL certificate retrieval](configuring-playbook-ssl-certificates.md) documentation page. + +## Increase logging verbosity + +```yaml +devture_traefik_config_log_level: DEBUG +``` + +## Disable access logs + +This will disable access logging. + +```yaml +devture_traefik_config_accessLog_enabled: false +``` + +## Enable Traefik Dashboard + +This will enable a Traefik [Dashboard](https://doc.traefik.io/traefik/operations/dashboard/) UI at `https://matrix.DOMAIN/dashboard/` (note the trailing `/`). + +```yaml +devture_traefik_dashboard_enabled: true +devture_traefik_dashboard_hostname: "{{ matrix_server_fqn_matrix }}" +devture_traefik_dashboard_basicauth_enabled: true +devture_traefik_dashboard_basicauth_user: YOUR_USERNAME_HERE +devture_traefik_dashboard_basicauth_password: YOUR_PASSWORD_HERE +``` + +## Additional configuration + +Use the `devture_traefik_configuration_extension_yaml` variable provided by the Traefik Ansible role to override or inject additional settings, even when no dedicated variable exists. + +```yaml +# This is a contrived example. +# You can enable and secure the Dashboard using dedicated variables. See above. +devture_traefik_configuration_extension_yaml: | + api: + dashboard: true +``` diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index c94977a2..b4bae786 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -59,9 +59,11 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Serving your base domain using this playbook's nginx server](configuring-playbook-base-domain-serving.md) (optional) -- [Configure Nginx](configuring-playbook-nginx.md) (optional, advanced) +- [Configure the Traefik reverse-proxy](configuring-playbook-traefik.md) (optional, advanced) -- [Using your own webserver, instead of this playbook's nginx proxy](configuring-playbook-own-webserver.md) (optional, advanced) +- (Deprecated) [Configure the Nginx reverse-proxy](configuring-playbook-nginx.md) (optional, advanced) + +- [Using your own webserver, instead of this playbook's default reverse-proxy](configuring-playbook-own-webserver.md) (optional, advanced) - [Adjusting TURN server configuration](configuring-playbook-turn.md) (optional, advanced) diff --git a/examples/vars.yml b/examples/vars.yml index 732d7eed..784bf061 100644 --- a/examples/vars.yml +++ b/examples/vars.yml @@ -21,6 +21,11 @@ matrix_homeserver_implementation: synapse # You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`). matrix_homeserver_generic_secret_key: '' +# By default, the playbook manages its own Traefik (https://doc.traefik.io/traefik/) reverse-proxy server. +# It will retrieve SSL certificates for you on-demand and forward requests to all other components. +# For alternatives, see `docs/configuring-playbook-own-webserver.md`. +matrix_playbook_reverse_proxy_type: playbook-managed-traefik + # This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains. # # In case SSL renewal fails at some point, you'll also get an email notification there. @@ -29,7 +34,7 @@ matrix_homeserver_generic_secret_key: '' # you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`). # # Example value: someone@example.com -matrix_ssl_lets_encrypt_support_email: '' +devture_traefik_config_certificatesResolvers_acme_email: '' # A Postgres password to use for the superuser Postgres user (called `matrix` by default). # diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 0f2e3b48..403af983 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -319,7 +319,7 @@ matrix_homeserver_app_service_config_files_auto: [] # - no nginx configuration will be dumped in /matrix/nginx/conf.d # - no port exposure will be done for any of the container services # - it's up to you to expose the ports you want, etc. -matrix_playbook_reverse_proxy_type: playbook-managed-nginx +matrix_playbook_reverse_proxy_type: '' matrix_playbook_service_host_bind_interface_prefix: "{{ '' if matrix_playbook_reverse_proxy_type not in ['other-nginx-non-container', 'other-on-same-host', 'other-on-another-host'] else ('0.0.0.0:' if matrix_playbook_reverse_proxy_type == 'other-on-another-host' else '127.0.0.1:') }}" diff --git a/roles/custom/matrix-base/tasks/validate_config.yml b/roles/custom/matrix-base/tasks/validate_config.yml index ab06ffcd..c663ab2e 100644 --- a/roles/custom/matrix-base/tasks/validate_config.yml +++ b/roles/custom/matrix-base/tasks/validate_config.yml @@ -54,6 +54,14 @@ msg: "Detected that variable matrix_architecture {{ matrix_architecture }} appears to be set incorrectly. See docs/alternative-architectures.md. Server appears to be {{ ansible_architecture }}." when: matrix_architecture not in ['amd64', 'arm32', 'arm64'] +- name: Fail if matrix_playbook_reverse_proxy_type is set incorrectly + ansible.builtin.fail: + msg: | + You have not explicitly specified the type of reverse-proxy you'd like to use (with a `matrix_playbook_reverse_proxy_type` variable in your configuration). + The playbook used to default to nginx (matrix-nginx-proxy), but things are changing with Traefik being the new default reverse-proxy. + Learn more here: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#traefik-is-the-default-reverse-proxy-now + when: matrix_playbook_reverse_proxy_type == '' + - name: Fail if matrix_playbook_reverse_proxy_type is set incorrectly ansible.builtin.fail: msg: "Detected that variable matrix_playbook_reverse_proxy_type (current value: `{{ matrix_playbook_reverse_proxy_type }}`) appears to be set incorrectly. See roles/custom/matrix-base/defaults/main.yml for valid choices." From 0452619820fd0bd98a6b9e20ced51b54bcf3a19a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 16:21:50 +0200 Subject: [PATCH 236/382] Add Related plabooks --- README.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/README.md b/README.md index dd446725..bd37d23e 100644 --- a/README.md +++ b/README.md @@ -188,3 +188,18 @@ When updating the playbook, refer to [the changelog](CHANGELOG.md) to catch up w - IRC channel: `#matrix-docker-ansible-deploy` on the [Libera Chat](https://libera.chat/) IRC network (irc.libera.chat:6697) - GitHub issues: [spantaleev/matrix-docker-ansible-deploy/issues](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues) + + +## Related + +You may also be interested in these other Ansible playbooks: + +- [gitea-docker-ansible-deploy](https://github.com/spantaleev/gitea-docker-ansible-deploy) - for deploying a [Gitea](https://gitea.io/) git version-control server + +- [nextcloud-docker-ansible-deploy](https://github.com/spantaleev/nextcloud-docker-ansible-deploy) - for deploying a [Nextcloud](https://nextcloud.com/) server + +- [peertube-docker-ansible-deploy](https://github.com/spantaleev/peertube-docker-ansible-deploy) - for deploying a [PeerTube](https://joinpeertube.org/) video-platform server + +- [vaultwarden-docker-ansible-deploy](https://github.com/spantaleev/vaultwarden-docker-ansible-deploy) - for deploying a [Vaultwarden](https://github.com/dani-garcia/vaultwarden) password manager server (unofficial [Bitwarden](https://bitwarden.com/) compatible server) + +They're all making use of Traefik as their reverse-proxy, so it should be easy to host all these services on the same server. Follow the `docs/configuring-playbook-interoperability.md` documentation in each playbook. From 7852d5eb26457b828fe8dc76327bb0639c36e3c7 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 26 Feb 2023 15:58:10 +0000 Subject: [PATCH 237/382] update mautrix-telegram 0.12.2 -> 0.13.0 --- roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml index 1221da84..4d040161 100644 --- a/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml @@ -17,7 +17,7 @@ matrix_mautrix_telegram_docker_repo: "https://mau.dev/mautrix/telegram.git" matrix_mautrix_telegram_docker_repo_version: "{{ 'master' if matrix_mautrix_telegram_version == 'latest' else matrix_mautrix_telegram_version }}" matrix_mautrix_telegram_docker_src_files_path: "{{ matrix_base_data_path }}/mautrix-telegram/docker-src" -matrix_mautrix_telegram_version: v0.12.2 +matrix_mautrix_telegram_version: v0.13.0 # See: https://mau.dev/mautrix/telegram/container_registry matrix_mautrix_telegram_docker_image: "{{ matrix_mautrix_telegram_docker_image_name_prefix }}mautrix/telegram:{{ matrix_mautrix_telegram_version }}" matrix_mautrix_telegram_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_telegram_container_image_self_build else 'dock.mau.dev/' }}" From ffff236167e8b228e6fe0a3fe2332057937fc9c7 Mon Sep 17 00:00:00 2001 From: Jan <31133207+Jaffex@users.noreply.github.com> Date: Sun, 26 Feb 2023 17:21:08 +0100 Subject: [PATCH 238/382] Update configuring-playbook-own-webserver.md Change documentation for host bind port when fronting traefik with another reverse proxy --- docs/configuring-playbook-own-webserver.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 3b9d822d..ff9d2c93 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -138,8 +138,8 @@ devture_traefik_container_web_host_bind_port: '127.0.0.1:81' devture_traefik_additional_entrypoints_auto: - name: matrix-federation - port: "{{ matrix_federation_public_port }}" - host_bind_port: "127.0.0.1:{{ matrix_federation_public_port }}" + port: 8449 + host_bind_port: '127.0.0.1:8449' config: {} ``` From 05b3f6565b4affa08fa71204088b27e288beb9d2 Mon Sep 17 00:00:00 2001 From: Jan <31133207+Jaffex@users.noreply.github.com> Date: Sun, 26 Feb 2023 17:21:59 +0100 Subject: [PATCH 239/382] Update CHANGELOG.md Fix typo in changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5cf821f3..cd592337 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -57,7 +57,7 @@ At some point in the **near** future (days, or even weeks at most), we hope to c ### How do I keep using my own other reverse-proxy? -We recommend that you follow the guide for [Fronting the integraed reverse-proxy webserver with another reverse-proxy](docs/configuring-playbook-own-webserver.md#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy). +We recommend that you follow the guide for [Fronting the integrated reverse-proxy webserver with another reverse-proxy](docs/configuring-playbook-own-webserver.md#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy). # 2023-02-25 From 7c622bd249821b1e4ddfb9fdd85c7cf43d74d45d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 19:12:10 +0200 Subject: [PATCH 240/382] Upgrade com.devture.ansible.role.traefik --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 56fa7701..f3ab0272 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.7-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: 5e3248cd4d17db9ff3ba90904e7330150dab599e + version: 2c949e88261c4398533ad16017475a1c75391487 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 From 31857ea9bb332463720bc638cb49d47fff96a223 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 19:49:01 +0200 Subject: [PATCH 241/382] Upgrade com.devture.ansible.role.traefik_certs_dumper --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index f3ab0272..fe0d85b5 100644 --- a/requirements.yml +++ b/requirements.yml @@ -52,4 +52,4 @@ version: 2c949e88261c4398533ad16017475a1c75391487 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git - version: e7563caa814e634d2f8deec4a41e970caa1c0ae4 + version: abeecc09f6fd165eff3cef5d6dd7696b9ca4ec27 From 9e4ecf4ac917d6d92116c297ab982e230dd50f3d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 22:05:36 +0200 Subject: [PATCH 242/382] Add matrix_dendrite_container_network/matrix_dendrite_container_additional_networks --- roles/custom/matrix-dendrite/defaults/main.yml | 8 ++++++++ roles/custom/matrix-dendrite/tasks/setup_install.yml | 5 +++++ .../dendrite/systemd/matrix-dendrite.service.j2 | 12 ++++++++++-- 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 525f4854..8c2dded4 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -33,6 +33,14 @@ matrix_dendrite_http_bind_address: "{{ (':' + matrix_dendrite_http_bind_port | s # This is passed as an `-https-bind-address` flag to the Dendrite server in the container matrix_dendrite_https_bind_address: "{{ (':' + matrix_dendrite_https_bind_port | string) if matrix_dendrite_https_bind_port else '' }}" +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_dendrite_container_network: "{{ matrix_docker_network }}" + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_dendrite_container_additional_networks: [] + # Controls whether the matrix-dendrite container exposes the HTTP port (tcp/{{ matrix_dendrite_http_bind_port }} in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:8008"), or empty string to not expose. diff --git a/roles/custom/matrix-dendrite/tasks/setup_install.yml b/roles/custom/matrix-dendrite/tasks/setup_install.yml index c516c5b7..3b8cbd9b 100644 --- a/roles/custom/matrix-dendrite/tasks/setup_install.yml +++ b/roles/custom/matrix-dendrite/tasks/setup_install.yml @@ -76,6 +76,11 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" +- name: Ensure Dendrite container network is created + community.general.docker_network: + name: "{{ matrix_dendrite_container_network }}" + driver: bridge + - name: Ensure matrix-dendrite.service installed ansible.builtin.template: src: "{{ role_path }}/templates/dendrite/systemd/matrix-dendrite.service.j2" diff --git a/roles/custom/matrix-dendrite/templates/dendrite/systemd/matrix-dendrite.service.j2 b/roles/custom/matrix-dendrite/templates/dendrite/systemd/matrix-dendrite.service.j2 index 4649c3a3..378cf596 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/systemd/matrix-dendrite.service.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/systemd/matrix-dendrite.service.j2 @@ -22,13 +22,15 @@ ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_s ExecStartPre={{ matrix_host_command_sleep }} 5 {% endif %} -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-dendrite \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-dendrite \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_dendrite_tmp_directory_size_mb }}m \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_dendrite_container_network }} \ {% if matrix_dendrite_container_http_host_bind_address and matrix_dendrite_http_bind_port %} -p {{ matrix_dendrite_container_http_host_bind_address }}:{{ matrix_dendrite_http_bind_port }} \ {% endif %} @@ -54,6 +56,12 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name -https-bind-address {{ matrix_dendrite_https_bind_address }} {% endif %} +{% for network in matrix_dendrite_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-dendrite +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-dendrite + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-dendrite 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-dendrite 2>/dev/null || true' ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-dendrite /bin/sh -c 'kill -HUP 1' From cdd23aafc6931aa9f6902a03f40fe0dc8e09d264 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 22:08:08 +0200 Subject: [PATCH 243/382] Add matrix_conduit_container_network/matrix_conduit_container_additional_networks --- roles/custom/matrix-conduit/defaults/main.yml | 8 ++++++ .../matrix-conduit/tasks/setup_install.yml | 26 ++++++++++++------- .../conduit/systemd/matrix-conduit.service.j2 | 12 +++++++-- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/roles/custom/matrix-conduit/defaults/main.yml b/roles/custom/matrix-conduit/defaults/main.yml index bbab5cde..7673a7d2 100644 --- a/roles/custom/matrix-conduit/defaults/main.yml +++ b/roles/custom/matrix-conduit/defaults/main.yml @@ -23,6 +23,14 @@ matrix_conduit_systemd_required_services_list: ["docker.service"] # List of systemd services that matrix-conduit.service wants matrix_conduit_systemd_wanted_services_list: [] +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_conduit_container_network: "{{ matrix_docker_network }}" + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_conduit_container_additional_networks: [] + # Extra arguments for the Docker container matrix_conduit_container_extra_arguments: [] diff --git a/roles/custom/matrix-conduit/tasks/setup_install.yml b/roles/custom/matrix-conduit/tasks/setup_install.yml index ef47c620..ecccbb32 100644 --- a/roles/custom/matrix-conduit/tasks/setup_install.yml +++ b/roles/custom/matrix-conduit/tasks/setup_install.yml @@ -1,14 +1,4 @@ --- -- name: Ensure Conduit Docker image is pulled - community.docker.docker_image: - name: "{{ matrix_conduit_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_conduit_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_conduit_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - name: Ensure Conduit config path exists ansible.builtin.file: @@ -34,6 +24,22 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" +- name: Ensure Conduit container network is created + community.general.docker_network: + name: "{{ matrix_conduit_container_network }}" + driver: bridge + +- name: Ensure Conduit container image is pulled + community.docker.docker_image: + name: "{{ matrix_conduit_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_conduit_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_conduit_docker_image_force_pull }}" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + - name: Ensure matrix-conduit.service installed ansible.builtin.template: src: "{{ role_path }}/templates/conduit/systemd/matrix-conduit.service.j2" diff --git a/roles/custom/matrix-conduit/templates/conduit/systemd/matrix-conduit.service.j2 b/roles/custom/matrix-conduit/templates/conduit/systemd/matrix-conduit.service.j2 index cdaead01..91c1e915 100644 --- a/roles/custom/matrix-conduit/templates/conduit/systemd/matrix-conduit.service.j2 +++ b/roles/custom/matrix-conduit/templates/conduit/systemd/matrix-conduit.service.j2 @@ -12,13 +12,15 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-conduit 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-conduit 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-conduit \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-conduit \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_conduit_tmp_directory_size_mb }}m \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_conduit_container_network }} \ --env CONDUIT_CONFIG=/etc/matrix-conduit/conduit.toml \ --mount type=bind,src={{ matrix_conduit_data_path }},dst=/var/lib/matrix-conduit \ --mount type=bind,src={{ matrix_conduit_config_path }},dst=/etc/matrix-conduit,ro \ @@ -27,6 +29,12 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% endfor %} {{ matrix_conduit_docker_image }} +{% for network in matrix_conduit_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-conduit +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-conduit + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-conduit 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-conduit 2>/dev/null || true' ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-conduit /bin/sh -c 'kill -HUP 1' From b84f25309b6b0351b5b24b5be2ca49c4c12195a6 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 22:09:37 +0200 Subject: [PATCH 244/382] Add matrix_homeserver_container_network --- group_vars/matrix_servers | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 359c6e3f..75076bbc 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -435,6 +435,15 @@ matrix_homeserver_container_federation_url: |- }[matrix_homeserver_implementation] }} +matrix_homeserver_container_network: |- + {{ + matrix_nginx_proxy_container_network if matrix_nginx_proxy_enabled else { + 'synapse': matrix_synapse_container_network, + 'dendrite': matrix_dendrite_container_network, + 'conduit': matrix_conduit_container_network, + }[matrix_homeserver_implementation] + }} + matrix_integration_manager_rest_url: "{{ matrix_dimension_integrations_rest_url if matrix_dimension_enabled else None }}" matrix_integration_manager_ui_url: "{{ matrix_dimension_integrations_ui_url if matrix_dimension_enabled else None }}" From 058a54fd0593994b25b421ede595e8a6d07b8183 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Feb 2023 23:06:36 +0200 Subject: [PATCH 245/382] Add native Traefik support to Dimension --- docs/configuring-playbook-dimension.md | 23 +++- group_vars/matrix_servers | 29 +++- .../custom/matrix-dimension/defaults/main.yml | 127 +++++++++++++++++- .../matrix-dimension/tasks/setup_install.yml | 13 ++ .../tasks/validate_config.yml | 33 +++++ .../matrix-dimension/templates/config.yaml.j2 | 14 +- .../matrix-dimension/templates/labels.j2 | 45 +++++++ .../systemd/matrix-dimension.service.j2 | 14 +- .../matrix-nginx-proxy/defaults/main.yml | 5 - .../matrix-nginx-proxy/templates/labels.j2 | 12 -- 10 files changed, 275 insertions(+), 40 deletions(-) create mode 100644 roles/custom/matrix-dimension/templates/labels.j2 diff --git a/docs/configuring-playbook-dimension.md b/docs/configuring-playbook-dimension.md index e201785a..4472e103 100644 --- a/docs/configuring-playbook-dimension.md +++ b/docs/configuring-playbook-dimension.md @@ -6,14 +6,29 @@ If you're just installing Matrix services for the first time, please continue wi **Note**: This playbook now supports running [Dimension](https://dimension.t2bot.io) in both a federated and [unfederated](https://github.com/turt2live/matrix-dimension/blob/master/docs/unfederated.md) environments. This is handled automatically based on the value of `matrix_synapse_federation_enabled`. Enabling Dimension, means that the `openid` API endpoints will be exposed on the Matrix Federation port (usually `8448`), even if [federation](configuring-playbook-federation.md) is disabled. It's something to be aware of, especially in terms of firewall whitelisting (make sure port `8448` is accessible). -## Prerequisites +## Decide on a domain and path -The `dimension.` DNS record must be created. See [Configuring your DNS server](configuring-dns.md) on how to set up DNS record correctly. +By default, Dimension is configured to use its own dedicated domain (`dimension.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). + +You can override the domain and path like this: + +```yaml +# Switch to another hostname compared to the default (`dimension.{{ matrix_domain }}`) +matrix_dimension_hostname: "integrations.{{ matrix_domain }}" + +``` + +While there is a `matrix_dimension_path_prefix` variable for changing the path where Dimension is served, overriding it is not possible right now due to [this Dimension issue](https://github.com/turt2live/matrix-dimension/issues/510). You must serve Dimension at a dedicated subdomain until this issue is solved. + + +## Adjusting DNS records + +Once you've decided on the domain and path, **you may need to adjust your DNS** records to point the Dimension domain to the Matrix server. ## Enable -[Dimension integrations manager](https://dimension.t2bot.io) installation is disabled by default. You can enable it in your configuration file (`inventory/host_vars/matrix./vars.yml`): +To enable Dimension, add this to your configuration file (`inventory/host_vars/matrix./vars.yml`): ```yaml matrix_dimension_enabled: true @@ -54,7 +69,7 @@ For more information on how to acquire an access token, visit [https://t2bot.io/ ## Installation -After these variables have been set, please run the following command to re-run setup and to restart Dimension: +After these variables have been set and you have potentially [adjusted your DNS records](#adjusting-dns-records), please run the following command to re-run setup and to restart Dimension: ``` ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 75076bbc..e9d244d2 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2154,12 +2154,32 @@ matrix_dimension_enabled: false matrix_dimension_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" -# Normally, matrix-nginx-proxy is enabled and nginx can reach Dimension over the container network. -# If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose -# the Dimension HTTP port to the local host. +matrix_dimension_scheme: "{{ 'https' if matrix_playbook_ssl_enabled else 'http' }}" + +matrix_dimension_hostname: "{{ matrix_server_fqn_dimension }}" + +matrix_dimension_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-dimension' }}" + +matrix_dimension_container_additional_networks: | + {{ + ( + ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + + + ([matrix_homeserver_container_network] if matrix_homeserver_container_network != matrix_dimension_container_network else []) + + + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != matrix_dimension_container_network else []) + ) | unique + }} + matrix_dimension_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '8184') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +matrix_dimension_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +matrix_dimension_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_dimension_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" + +matrix_dimension_homeserver_clientServerUrl: "{{ matrix_homeserver_container_url }}" matrix_dimension_homeserver_federationUrl: "{{ matrix_homeserver_container_federation_url }}" +matrix_dimension_homeserver_mediaUrl: "https://{{ matrix_server_fqn_matrix }}" matrix_dimension_systemd_required_services_list: | {{ @@ -2443,7 +2463,7 @@ matrix_nginx_proxy_proxy_element_enabled: "{{ matrix_client_element_enabled and matrix_nginx_proxy_proxy_hydrogen_enabled: "{{ matrix_client_hydrogen_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" -matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" +matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" @@ -2460,7 +2480,6 @@ matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true -matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" diff --git a/roles/custom/matrix-dimension/defaults/main.yml b/roles/custom/matrix-dimension/defaults/main.yml index 4be76fd1..e66f9009 100644 --- a/roles/custom/matrix-dimension/defaults/main.yml +++ b/roles/custom/matrix-dimension/defaults/main.yml @@ -3,6 +3,15 @@ matrix_dimension_enabled: false +matrix_dimension_scheme: https + +# The hostname at which Dimension is served. +matrix_dimension_hostname: '' + +# The path at which Dimension is exposed. +# This value must either be `/` or not end with a slash (e.g. `/dimension`). +matrix_dimension_path_prefix: / + # You are required to specify an access token for Dimension to work. # For information on how to acquire an access token, visit https://t2bot.io/docs/access_tokens matrix_dimension_access_token: "" @@ -35,21 +44,129 @@ matrix_dimension_systemd_wanted_services_list: [] matrix_dimension_user_uid: '1000' matrix_dimension_user_gid: '1000' +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_dimension_container_network: matrix-dimension + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_dimension_container_additional_networks: [] + # Controls whether the matrix-dimension container exposes its HTTP port (tcp/8184 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:8184"), or empty string to not expose. matrix_dimension_container_http_host_bind_port: '' + +# matrix_dimension_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_dimension_container_labels_additional_labels`. +matrix_dimension_container_labels_traefik_enabled: true +matrix_dimension_container_labels_traefik_docker_network: "{{ matrix_dimension_container_network }}" +matrix_dimension_container_labels_traefik_hostname: "{{ matrix_dimension_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/dimension`). +matrix_dimension_container_labels_traefik_path_prefix: "{{ matrix_dimension_path_prefix }}" +matrix_dimension_container_labels_traefik_rule: "Host(`{{ matrix_dimension_container_labels_traefik_hostname }}`){% if matrix_dimension_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_dimension_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_dimension_container_labels_traefik_priority: 0 +matrix_dimension_container_labels_traefik_entrypoints: web-secure +matrix_dimension_container_labels_traefik_tls: "{{ matrix_dimension_container_labels_traefik_entrypoints != 'web' }}" +matrix_dimension_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_dimension_container_labels_traefik_additional_response_headers_custom` +matrix_dimension_container_labels_traefik_additional_response_headers: "{{ matrix_dimension_container_labels_traefik_additional_response_headers_auto | combine(matrix_dimension_container_labels_traefik_additional_response_headers_custom) }}" +matrix_dimension_container_labels_traefik_additional_response_headers_auto: | + {{ + {} + | combine ({'X-XSS-Protection': matrix_dimension_http_header_xss_protection} if matrix_dimension_http_header_xss_protection else {}) + | combine ({'X-Frame-Options': matrix_dimension_http_header_frame_options} if matrix_dimension_http_header_frame_options else {}) + | combine ({'X-Content-Type-Options': matrix_dimension_http_header_content_type_options} if matrix_dimension_http_header_content_type_options else {}) + | combine ({'Content-Security-Policy': matrix_dimension_http_header_content_security_policy} if matrix_dimension_http_header_content_security_policy else {}) + | combine ({'Permission-Policy': matrix_dimension_http_header_content_permission_policy} if matrix_dimension_http_header_content_permission_policy else {}) + | combine ({'Strict-Transport-Security': matrix_dimension_http_header_strict_transport_security} if matrix_dimension_http_header_strict_transport_security and matrix_dimension_container_labels_traefik_tls else {}) + }} +matrix_dimension_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_dimension_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_dimension_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_dimension_container_labels_additional_labels: '' + + # A list of extra arguments to pass to the container matrix_dimension_container_extra_arguments: [] -matrix_dimension_integrations_ui_url: "https://{{ matrix_server_fqn_dimension }}/element" -matrix_dimension_integrations_rest_url: "https://{{ matrix_server_fqn_dimension }}/api/v1/scalar" -matrix_dimension_integrations_widgets_urls: ["https://{{ matrix_server_fqn_dimension }}/widgets"] -matrix_dimension_integrations_jitsi_widget_url: "https://{{ matrix_server_fqn_dimension }}/widgets/jitsi" +# Public facing base URL of the Dimension service. +# It should not end with a slash. +matrix_dimension_base_url: "{{ matrix_dimension_scheme }}://{{ matrix_dimension_hostname }}{{ '' if matrix_dimension_path_prefix == '/' else matrix_dimension_path_prefix }}" + +matrix_dimension_integrations_ui_url: "{{ matrix_dimension_base_url }}/element" +matrix_dimension_integrations_rest_url: "{{ matrix_dimension_base_url }}/api/v1/scalar" +matrix_dimension_integrations_widgets_urls: ["{{ matrix_dimension_base_url }}/widgets"] +matrix_dimension_integrations_jitsi_widget_url: "{{ matrix_dimension_base_url }}/widgets/jitsi" + +matrix_dimension_homeserver_clientServerUrl: '' # noqa var-naming +matrix_dimension_homeserver_federationUrl: '' # noqa var-naming +matrix_dimension_homeserver_mediaUrl: '' # noqa var-naming -matrix_dimension_homeserver_federationUrl: "" # noqa var-naming +# Controls the dimension.publicUrl configuration option +matrix_dimension_dimension_publicUrl: "{{ matrix_dimension_base_url }}" # noqa var-naming +# Specifies the value of the `X-XSS-Protection` header +# Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. +# +# Learn more about it is here: +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection +# - https://portswigger.net/web-security/cross-site-scripting/reflected +matrix_dimension_http_header_xss_protection: "1; mode=block" + +# Specifies the value of the `X-Frame-Options` header which controls whether framing can happen. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options +matrix_dimension_http_header_frame_options: '' + +# Specifies the value of the `X-Content-Type-Options` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options +matrix_dimension_http_header_content_type_options: nosniff + +# Specifies the value of the `Content-Security-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy +matrix_dimension_http_header_content_security_policy: '' + +# Specifies the value of the `Permission-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy +matrix_dimension_http_header_content_permission_policy: "{{ 'interest-cohort=()' if matrix_dimension_floc_optout_enabled else '' }}" + +# Specifies the value of the `Strict-Transport-Security` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +matrix_dimension_http_header_strict_transport_security: "max-age=31536000; includeSubDomains{{ '; preload' if matrix_dimension_hsts_preload_enabled else '' }}" + +# Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses +# +# Learn more about what it is here: +# - https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea +# - https://paramdeo.com/blog/opting-your-website-out-of-googles-floc-network +# - https://amifloced.org/ +# +# Of course, a better solution is to just stop using browsers (like Chrome), which participate in such tracking practices. +# See: `matrix_dimension_content_permission_policy` +matrix_dimension_floc_optout_enabled: true + +# Controls if HSTS preloading is enabled +# +# In its strongest and recommended form, the [HSTS policy](https://www.chromium.org/hsts) includes all subdomains, and +# indicates a willingness to be "preloaded" into browsers: +# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload` +# For more information visit: +# - https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +# - https://hstspreload.org/#opt-in +# See: `matrix_dimension_http_header_strict_transport_security` +matrix_dimension_hsts_preload_enabled: false # Database-related configuration fields. # diff --git a/roles/custom/matrix-dimension/tasks/setup_install.yml b/roles/custom/matrix-dimension/tasks/setup_install.yml index 26617f7a..b1f0c242 100644 --- a/roles/custom/matrix-dimension/tasks/setup_install.yml +++ b/roles/custom/matrix-dimension/tasks/setup_install.yml @@ -85,6 +85,14 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_dimension_user_gid }}" +- name: Ensure Dimension labels installed + ansible.builtin.template: + src: "{{ role_path }}/templates/labels.j2" + dest: "{{ matrix_dimension_base_path }}/labels" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + - name: Ensure Dimension image is pulled community.docker.docker_image: name: "{{ matrix_dimension_docker_image }}" @@ -120,6 +128,11 @@ pull: true when: "matrix_dimension_container_image_self_build | bool" +- name: Ensure Dimension container network is created + community.general.docker_network: + name: "{{ matrix_dimension_container_network }}" + driver: bridge + - name: Ensure matrix-dimension.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-dimension.service.j2" diff --git a/roles/custom/matrix-dimension/tasks/validate_config.yml b/roles/custom/matrix-dimension/tasks/validate_config.yml index 8f681e08..b001ceed 100644 --- a/roles/custom/matrix-dimension/tasks/validate_config.yml +++ b/roles/custom/matrix-dimension/tasks/validate_config.yml @@ -6,7 +6,12 @@ You need to define a required configuration setting (`{{ item.name }}`). when: "item.when | bool and vars[item.name] == ''" with_items: + - {'name': 'matrix_dimension_hostname', when: true} + - {'name': 'matrix_dimension_container_network', when: true} - {'name': 'matrix_dimension_access_token', when: true} + - {'name': 'matrix_dimension_homeserver_clientServerUrl', when: true} + - {'name': 'matrix_dimension_homeserver_federationUrl', when: true} + - {'name': 'matrix_dimension_homeserver_mediaUrl', when: true} - {'name': 'matrix_dimension_database_hostname', when: "{{ matrix_dimension_database_engine == 'postgres' }}"} - name: (Deprecation) Catch and report renamed Dimension variables @@ -17,3 +22,31 @@ when: "item.old in vars" with_items: - {'old': 'matrix_dimension_container_expose_port', 'new': ''} + +- when: matrix_dimension_container_labels_traefik_enabled | bool + block: + - name: Fail if required Dimension Traefik settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_dimension_container_labels_traefik_hostname + - matrix_dimension_container_labels_traefik_path_prefix + + # We ensure it doesn't end with a slash, because we handle both (slash and no-slash). + # Knowing that `matrix_dimension_container_labels_traefik_path_prefix` does not end with a slash + # ensures we know how to set these routes up without having to do "does it end with a slash" checks elsewhere. + - name: Fail if matrix_dimension_container_labels_traefik_path_prefix ends with a slash + ansible.builtin.fail: + msg: >- + matrix_dimension_container_labels_traefik_path_prefix (`{{ matrix_dimension_container_labels_traefik_path_prefix }}`) must either be `/` or not end with a slash (e.g. `/dimension`). + when: "matrix_dimension_container_labels_traefik_path_prefix != '/' and matrix_dimension_container_labels_traefik_path_prefix[-1] == '/'" + + - name: Fail if matrix_dimension_container_labels_traefik_path_prefix is not / + ansible.builtin.fail: + msg: >- + matrix_dimension_container_labels_traefik_path_prefix (`{{ matrix_dimension_container_labels_traefik_path_prefix }}`) must currently be set to `/`. + Other values are not supported. + See: https://github.com/turt2live/matrix-dimension/issues/510 + when: "matrix_dimension_container_labels_traefik_path_prefix != '/' and matrix_dimension_container_labels_traefik_path_prefix[-1] == '/'" diff --git a/roles/custom/matrix-dimension/templates/config.yaml.j2 b/roles/custom/matrix-dimension/templates/config.yaml.j2 index 592c65ac..c387c91b 100644 --- a/roles/custom/matrix-dimension/templates/config.yaml.j2 +++ b/roles/custom/matrix-dimension/templates/config.yaml.j2 @@ -9,30 +9,30 @@ web: homeserver: # The domain name of the homeserver. This is used in many places, such as with go-neb # setups, to identify the homeserver. - name: "{{ matrix_domain }}" + name: {{ matrix_domain | to_json }} # The URL that Dimension, go-neb, and other services provisioned by Dimension should # use to access the homeserver with. - clientServerUrl: "{{ matrix_homeserver_container_url }}" + clientServerUrl: {{ matrix_dimension_homeserver_clientServerUrl | to_json }} # The URL that Dimension should use when trying to communicate with federated APIs on # the homeserver. If not supplied or left empty Dimension will try to resolve the address # through the normal federation process. - federationUrl: "{{ matrix_dimension_homeserver_federationUrl }}" + federationUrl: {{ matrix_dimension_homeserver_federationUrl | to_json }} # The URL that Dimension will redirect media requests to for downloading media such as # stickers. If not supplied or left empty Dimension will use the clientServerUrl. - mediaUrl: "https://{{ matrix_server_fqn_matrix }}" + mediaUrl: {{ matrix_dimension_homeserver_mediaUrl | to_json }} # The access token Dimension should use for miscellaneous access to the homeserver. This # should be for a user on the configured homeserver: any user will do, however it is # recommended to use a dedicated user (such as @dimension:t2bot.io). For information on # how to acquire an access token, visit https://t2bot.io/docs/access_tokens - accessToken: "{{ matrix_dimension_access_token }}" + accessToken: {{ matrix_dimension_access_token | to_json }} # These users can modify the integrations this Dimension supports. # To access the admin interface, open Dimension in Element and click the settings icon. -admins: {{ matrix_dimension_admins|to_json }} +admins: {{ matrix_dimension_admins | to_json }} # IPs and CIDR ranges listed here will be blocked from being widgets. # Note: Widgets may still be embedded with restricted content, although not through Dimension directly. @@ -72,4 +72,4 @@ goneb: dimension: # This is where Dimension is accessible from clients. Be sure to set this # to your own Dimension instance. - publicUrl: "https://{{ matrix_server_fqn_dimension }}" + publicUrl: {{ matrix_dimension_dimension_publicUrl | to_json }} diff --git a/roles/custom/matrix-dimension/templates/labels.j2 b/roles/custom/matrix-dimension/templates/labels.j2 new file mode 100644 index 00000000..24b570cb --- /dev/null +++ b/roles/custom/matrix-dimension/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_dimension_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_dimension_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_dimension_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_dimension_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-dimension-slashless-redirect.redirectregex.regex=({{ matrix_dimension_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-dimension-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-dimension-slashless-redirect'] %} +{% endif %} + +{% if matrix_dimension_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-dimension-strip-prefix.stripprefix.prefixes={{ matrix_dimension_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-dimension-strip-prefix'] %} +{% endif %} + +{% if matrix_dimension_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_dimension_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-dimension-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-dimension-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-dimension.rule={{ matrix_dimension_container_labels_traefik_rule }} +{% if matrix_dimension_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-dimension.priority={{ matrix_dimension_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-dimension.service=matrix-dimension +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-dimension.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-dimension.entrypoints={{ matrix_dimension_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-dimension.tls={{ matrix_dimension_container_labels_traefik_tls | to_json }} +{% if matrix_dimension_container_labels_traefik_tls %} +traefik.http.routers.matrix-dimension.tls.certResolver={{ matrix_dimension_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-dimension.loadbalancer.server.port=8184 +{% endif %} + +{{ matrix_dimension_container_labels_additional_labels }} diff --git a/roles/custom/matrix-dimension/templates/systemd/matrix-dimension.service.j2 b/roles/custom/matrix-dimension/templates/systemd/matrix-dimension.service.j2 index 07428ffa..0b14a117 100644 --- a/roles/custom/matrix-dimension/templates/systemd/matrix-dimension.service.j2 +++ b/roles/custom/matrix-dimension/templates/systemd/matrix-dimension.service.j2 @@ -21,25 +21,35 @@ ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_s ExecStartPre=-{{ matrix_host_command_chown }} {{ matrix_dimension_user_uid }}:{{ matrix_dimension_user_gid }} {{ matrix_dimension_sqlite_database_path_local }} {% endif %} -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-dimension \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-dimension \ --log-driver=none \ --user={{ matrix_dimension_user_uid }}:{{ matrix_dimension_user_gid }} \ --cap-drop=ALL \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_dimension_container_network }} \ {% if matrix_dimension_widgets_allow_self_signed_ssl_certificates %} -e NODE_TLS_REJECT_UNAUTHORIZED=0 \ {% endif %} {% if matrix_dimension_container_http_host_bind_port %} -p {{ matrix_dimension_container_http_host_bind_port }}:8184 \ {% endif %} + --label-file={{ matrix_dimension_base_path }}/labels \ --mount type=bind,src={{ matrix_dimension_base_path }},dst=/data \ {% for arg in matrix_dimension_container_extra_arguments %} {{ arg }} \ {% endfor %} {{ matrix_dimension_docker_image }} +{% for network in matrix_dimension_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-dimension +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-dimension + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-dimension 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-dimension 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-dimension diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index a098a098..f04f19f4 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -61,11 +61,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule: "Host(`{{ matrix_ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" -matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname: "{{ matrix_server_fqn_dimension }}" -matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_dimension_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_hostname: "{{ matrix_server_fqn_etherpad }}" matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 6afa6029..b4c68471 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -39,18 +39,6 @@ traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints={{ matrix_ {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_dimension_enabled %} -# Dimension -traefik.http.routers.matrix-nginx-proxy-dimension.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_rule }} -traefik.http.routers.matrix-nginx-proxy-dimension.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-dimension.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_dimension_tls %} -traefik.http.routers.matrix-nginx-proxy-dimension.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-dimension.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - {% if matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled %} # Etherpad traefik.http.routers.matrix-nginx-proxy-etherpad.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_rule }} From d5910d0421f8debb1939c2246916271e0d91b27d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 08:27:41 +0200 Subject: [PATCH 246/382] Upgrade com.devture.ansible.role.docker_sdk_for_python Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2522 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index fe0d85b5..d38383dc 100644 --- a/requirements.yml +++ b/requirements.yml @@ -4,7 +4,7 @@ version: 6.1.0 - src: git+https://github.com/devture/com.devture.ansible.role.docker_sdk_for_python.git - version: 54d6d1831f063e4a99b66c722f49053d59f8ee60 + version: 129c8590e106b83e6f4c259649a613c6279e937a - src: git+https://github.com/devture/com.devture.ansible.role.playbook_help.git version: c1f40e82b4d6b072b6f0e885239322bdaaaf554f From d544f746e8f2da3d01c9e00d699ec8e255b5bc9c Mon Sep 17 00:00:00 2001 From: Thom Wiggers Date: Mon, 27 Feb 2023 13:40:23 +0100 Subject: [PATCH 247/382] Update irc appservice --- roles/custom/matrix-bridge-appservice-irc/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-appservice-irc/defaults/main.yml b/roles/custom/matrix-bridge-appservice-irc/defaults/main.yml index 5e8c1191..ce39bc56 100644 --- a/roles/custom/matrix-bridge-appservice-irc/defaults/main.yml +++ b/roles/custom/matrix-bridge-appservice-irc/defaults/main.yml @@ -11,7 +11,7 @@ matrix_appservice_irc_docker_src_files_path: "{{ matrix_base_data_path }}/appser # matrix_appservice_irc_version used to contain the full Docker image tag (e.g. `release-X.X.X`). # It's a bare version number now. We try to somewhat retain compatibility below. -matrix_appservice_irc_version: 0.36.0 +matrix_appservice_irc_version: 0.37.0 matrix_appservice_irc_docker_image: "{{ matrix_container_global_registry_prefix }}matrixdotorg/matrix-appservice-irc:{{ matrix_appservice_irc_docker_image_tag }}" matrix_appservice_irc_docker_image_tag: "{{ 'latest' if matrix_appservice_irc_version == 'latest' else ('release-' + matrix_appservice_irc_version) }}" matrix_appservice_irc_docker_image_force_pull: "{{ matrix_appservice_irc_docker_image.endswith(':latest') }}" From 724373b12358664dd226a658ed2ff4c7f9d21d6c Mon Sep 17 00:00:00 2001 From: Jost Alemann Date: Mon, 27 Feb 2023 14:59:36 +0100 Subject: [PATCH 248/382] add nginx to list of own webserver example configs --- docs/configuring-playbook-own-webserver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index ff9d2c93..35c22b35 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -232,4 +232,4 @@ Some of these services are configured with certain default expecations with rega For each new playbook service that you enable, you'll need special handling. -The [`examples/`](../examples/) directory contains examples for various servers: Caddy, Apache, HAproxy, etc. +The [`examples/`](../examples/) directory contains examples for various servers: Caddy, Apache, HAproxy, Nginx, etc. From e6ba7cc2c915d4f0674da72aa911e140a9390e4c Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 16:32:37 +0200 Subject: [PATCH 249/382] Upgrade com.devture.ansible.role.traefik --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index d38383dc..c882aee9 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.7-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: 2c949e88261c4398533ad16017475a1c75391487 + version: acc49a8059206743f1921f94891ebc177bf79bd2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: abeecc09f6fd165eff3cef5d6dd7696b9ca4ec27 From 8dacdb038b67dc41711bf72648f749a106c550e6 Mon Sep 17 00:00:00 2001 From: Jost Alemann Date: Mon, 27 Feb 2023 15:34:37 +0100 Subject: [PATCH 250/382] add: nginx example conf + readme for fronting playbooks traefik with own nginx --- examples/nginx/README.md | 32 +++++++++++++ examples/nginx/matrix.conf | 96 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 128 insertions(+) create mode 100644 examples/nginx/README.md create mode 100644 examples/nginx/matrix.conf diff --git a/examples/nginx/README.md b/examples/nginx/README.md new file mode 100644 index 00000000..d2d2a6ce --- /dev/null +++ b/examples/nginx/README.md @@ -0,0 +1,32 @@ +# Nginx reverse-proxy fronting playbook's Traefik + +This directory contains a sample config that show you how to do reverse-proxying using Nginx and the playbook's internal traefik container. + +This is for when you wish to front the playbook's integrated traefik container with a self-managed Nginx reverse-proxy running on the same server. +See the [Using your own webserver, instead of this playbook's nginx proxy & Fronting the integrated reverse-proxy webserver with another reverse-proxy](../../docs/configuring-playbook-own-webserver.md#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy) documentation page and follow the instructions for the playbook's configuration (`inventory/host_vars/matrix./vars.yml`). + +That is this part: +**For Traefik** fronted by another reverse-proxy, you would need some configuration like this: + +```yaml +matrix_playbook_reverse_proxy_type: playbook-managed-traefik + +# Ensure that public urls use https +matrix_playbook_ssl_enabled: true + +# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval +devture_traefik_config_entrypoint_web_secure_enabled: false + +devture_traefik_container_web_host_bind_port: '127.0.0.1:81' + +devture_traefik_additional_entrypoints_auto: + - name: matrix-federation + port: 8449 + host_bind_port: '127.0.0.1:8449' + config: {} +``` + +**NOTE**: +- that this also disables SSL certificate retrieval, which then has to be done manually (e.g. by using certbot and setting the appropriate path as found in [the example nginx configuration file](./matrix.conf)). For the example nginx config one certificate is used that contains all the used subdomains. +- that [the example nginx configuration file](./matrix.conf) has to be adapted to whatever services you are using. For example, remove element.domain.com from the `server_name` list if you don't use Element web client or add dimension.domain.com to it if you do use Dimension. +- that this is just an example and may not be entirely accurate. It may also not cover other use cases (enabling various services or bridges requires additional reverse-proxying configuration). diff --git a/examples/nginx/matrix.conf b/examples/nginx/matrix.conf new file mode 100644 index 00000000..366a8a8e --- /dev/null +++ b/examples/nginx/matrix.conf @@ -0,0 +1,96 @@ +server { + listen 443 ssl http2; + listen [::]:443 ssl http2; + + # TODO: add/remove services and their subdomains if you use/don't use them + # this example is using hosting something on the base domain and an element web client, so example.com and element.example.com are listed in addition to matrix.example.com + # if you don't use those, you can remove them + # if you use e.g. dimension on dimension.example.com, add dimension.example.com to the server_name list + server_name example.com matrix.example.com element.example.com; + + location / { + # note: do not add a path (even a single /) after the port in `proxy_pass`, + # otherwise, nginx will canonicalise the URI and cause signature verification + # errors. + proxy_pass http://localhost:81; + proxy_set_header X-Forwarded-For $remote_addr; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + + access_log /var/log/nginx/matrix.access.log; + error_log /var/log/nginx/matrix.error.log; + + # Nginx by default only allows file uploads up to 1M in size + # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml + client_max_body_size 50M; + } + + # TODO: adapt the path to your ssl certificate for the domains listed on server_name + ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot + # TODO: adapt the path to your ssl certificate for the domains listed on server_name + ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot +} + +# settings for matrix federation +server { + # For the federation port + listen 8448 ssl http2 default_server; + listen [::]:8448 ssl http2 default_server; + + server_name matrix.example.com; + + location / { + proxy_pass http://localhost:8449; + proxy_set_header X-Forwarded-For $remote_addr; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $host; + + access_log /var/log/nginx/matrix.access.log; + error_log /var/log/nginx/matrix.error.log; + + # Nginx by default only allows file uploads up to 1M in size + # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml + client_max_body_size 50M; + } + # TODO: adapt the path to your ssl certificate for the domains listed on server_name + ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot + # TODO: adapt the path to your ssl certificate for the domains listed on server_name + ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot + include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot +} + +# ensure using https +# TODO: remove server blocks that you don't use / add server blocks for domains you do use +server { + if ($host = example.com) { + return 301 https://$host$request_uri; + } # managed by Certbot + + server_name example.com; + listen 80; + return 404; # managed by Certbot +} + +server { + if ($host = matrix.example.com) { + return 301 https://$host$request_uri; + } # managed by Certbot + + server_name matrix.example.com; + listen 80; + return 404; # managed by Certbot +} + +server { + if ($host = element.example.com) { + return 301 https://$host$request_uri; + } # managed by Certbot + + server_name element.example.com; + listen 80; + return 404; # managed by Certbot +} \ No newline at end of file From 89b9fca85cbe88673cf9ff63bce5656be6393ca6 Mon Sep 17 00:00:00 2001 From: Jost Alemann Date: Mon, 27 Feb 2023 15:51:04 +0100 Subject: [PATCH 251/382] remove duplicate of playbook config instructions --- examples/nginx/README.md | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/examples/nginx/README.md b/examples/nginx/README.md index d2d2a6ce..038b060e 100644 --- a/examples/nginx/README.md +++ b/examples/nginx/README.md @@ -5,27 +5,6 @@ This directory contains a sample config that show you how to do reverse-proxying This is for when you wish to front the playbook's integrated traefik container with a self-managed Nginx reverse-proxy running on the same server. See the [Using your own webserver, instead of this playbook's nginx proxy & Fronting the integrated reverse-proxy webserver with another reverse-proxy](../../docs/configuring-playbook-own-webserver.md#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy) documentation page and follow the instructions for the playbook's configuration (`inventory/host_vars/matrix./vars.yml`). -That is this part: -**For Traefik** fronted by another reverse-proxy, you would need some configuration like this: - -```yaml -matrix_playbook_reverse_proxy_type: playbook-managed-traefik - -# Ensure that public urls use https -matrix_playbook_ssl_enabled: true - -# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval -devture_traefik_config_entrypoint_web_secure_enabled: false - -devture_traefik_container_web_host_bind_port: '127.0.0.1:81' - -devture_traefik_additional_entrypoints_auto: - - name: matrix-federation - port: 8449 - host_bind_port: '127.0.0.1:8449' - config: {} -``` - **NOTE**: - that this also disables SSL certificate retrieval, which then has to be done manually (e.g. by using certbot and setting the appropriate path as found in [the example nginx configuration file](./matrix.conf)). For the example nginx config one certificate is used that contains all the used subdomains. - that [the example nginx configuration file](./matrix.conf) has to be adapted to whatever services you are using. For example, remove element.domain.com from the `server_name` list if you don't use Element web client or add dimension.domain.com to it if you do use Dimension. From 84e6677bc0fcbde82a91db920b9a0af84b0566c8 Mon Sep 17 00:00:00 2001 From: Jost Alemann Date: Mon, 27 Feb 2023 15:56:34 +0100 Subject: [PATCH 252/382] in own webserver doc link to nginx fronting traefik example --- docs/configuring-playbook-own-webserver.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 35c22b35..cae1486e 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -143,6 +143,8 @@ devture_traefik_additional_entrypoints_auto: config: {} ``` +For an example where the playbook's traefik is fronted by Nginx running on the same server, see [Nginx reverse-proxy fronting playbook's Traefik](../examples/nginx/README.md). + (Deprecated) **For `matrix-nginx-proxy`** fronted by another reverse-proxy, you would need some configuration like this: ```yaml From 9e7415afa269d84dbfab98f7ff16eed371a56d63 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 17:07:44 +0200 Subject: [PATCH 253/382] Ensure Buscarron is part of the Postgres network --- group_vars/matrix_servers | 2 ++ 1 file changed, 2 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index e9d244d2..40daa42e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1707,6 +1707,8 @@ matrix_bot_buscarron_container_additional_networks: | ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + ([matrix_nginx_proxy_container_network] if matrix_nginx_proxy_enabled and matrix_bot_buscarron_container_network != matrix_nginx_proxy_container_network else []) + + + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != matrix_bot_buscarron_container_network else []) ) | unique }} From b388a01ab7df38da612e9fceaa23ba8b23289b38 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 17:09:19 +0200 Subject: [PATCH 254/382] Wire all certResolver variables to devture_traefik_certResolver_primary This allows people wishing to change or unset the resolver, to have a single variable which they can toggle. Unsetting the resolver is useful for using your own certificates (not coming from a certificate resolver). --- group_vars/matrix_servers | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 40daa42e..4827a3c2 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1715,6 +1715,7 @@ matrix_bot_buscarron_container_additional_networks: | matrix_bot_buscarron_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" matrix_bot_buscarron_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_bot_buscarron_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_bot_buscarron_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" ###################################################################### # @@ -2051,6 +2052,7 @@ matrix_rageshake_container_http_host_bind_port: "{{ (matrix_playbook_service_hos matrix_rageshake_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" matrix_rageshake_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_rageshake_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_rageshake_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" ###################################################################### # @@ -2178,6 +2180,7 @@ matrix_dimension_container_http_host_bind_port: "{{ (matrix_playbook_service_hos matrix_dimension_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" matrix_dimension_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_dimension_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_dimension_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" matrix_dimension_homeserver_clientServerUrl: "{{ matrix_homeserver_container_url }}" matrix_dimension_homeserver_federationUrl: "{{ matrix_homeserver_container_federation_url }}" @@ -2480,6 +2483,7 @@ matrix_nginx_proxy_proxy_ntfy_enabled: "{{ ntfy_enabled and matrix_playbook_reve matrix_nginx_proxy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" matrix_nginx_proxy_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_nginx_proxy_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" @@ -2971,6 +2975,7 @@ matrix_sygnal_container_http_host_bind_port: "{{ (matrix_playbook_service_host_b matrix_sygnal_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" matrix_sygnal_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_sygnal_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_sygnal_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" ###################################################################### # @@ -3004,6 +3009,7 @@ ntfy_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_inter ntfy_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" ntfy_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" ntfy_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +ntfy_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" ntfy_visitor_request_limit_exempt_hosts_hostnames_auto: | {{ @@ -3061,6 +3067,7 @@ matrix_client_element_container_additional_networks: "{{ [matrix_playbook_revers matrix_client_element_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_client_element_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_client_element_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_client_element_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" matrix_client_element_default_hs_url: "{{ matrix_homeserver_url }}" matrix_client_element_default_is_url: "{{ matrix_identity_server_url }}" @@ -3114,6 +3121,7 @@ matrix_client_hydrogen_container_additional_networks: "{{ [matrix_playbook_rever matrix_client_hydrogen_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_client_hydrogen_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_client_hydrogen_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_client_hydrogen_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" matrix_client_hydrogen_default_hs_url: "{{ matrix_homeserver_url }}" @@ -3147,6 +3155,7 @@ matrix_client_cinny_container_additional_networks: "{{ [matrix_playbook_reverse_ matrix_client_cinny_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_client_cinny_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_client_cinny_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_client_cinny_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" matrix_client_cinny_default_hs_url: "{{ matrix_homeserver_url }}" @@ -3326,6 +3335,7 @@ matrix_synapse_admin_container_additional_networks: "{{ [matrix_playbook_reverse matrix_synapse_admin_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" matrix_synapse_admin_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" matrix_synapse_admin_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_synapse_admin_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" ###################################################################### # @@ -3357,6 +3367,7 @@ prometheus_node_exporter_container_additional_networks: "{{ [matrix_playbook_rev prometheus_node_exporter_container_labels_traefik_enabled: false prometheus_node_exporter_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" prometheus_node_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +prometheus_node_exporter_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" ###################################################################### # @@ -3388,6 +3399,7 @@ prometheus_postgres_exporter_container_additional_networks: "{{ [matrix_playbook prometheus_postgres_exporter_container_labels_traefik_enabled: false prometheus_postgres_exporter_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" prometheus_postgres_exporter_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +prometheus_postgres_exporter_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" prometheus_postgres_exporter_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" prometheus_postgres_exporter_database_username: matrix_prometheus_postgres_exporter @@ -3491,6 +3503,7 @@ grafana_container_additional_networks: | grafana_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" grafana_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" grafana_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +grafana_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" # Normally, matrix-nginx-proxy is enabled and nginx can reach Grafana over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose From 7716c306f1d9a99fd3302fb04ac535d643be2e60 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 17:26:22 +0200 Subject: [PATCH 255/382] Split Traefik rules for Matrix Client and Matrix Federation APIs This helps people who run federation on another domain (etc.). They should now easily be able to add a custom rule for federation traffic. --- roles/custom/matrix-nginx-proxy/defaults/main.yml | 5 ++++- roles/custom/matrix-nginx-proxy/tasks/validate_config.yml | 2 ++ roles/custom/matrix-nginx-proxy/templates/labels.j2 | 4 ++-- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index f04f19f4..6565e1b9 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -57,7 +57,10 @@ matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_rule: "Host(`{{ ma matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_matrix_hostname: "{{ matrix_server_fqn_matrix }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_hostname }}`)" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_client_hostname: "{{ matrix_server_fqn_matrix }}" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_client_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_client_hostname }}`)" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_hostname: "{{ matrix_server_fqn_matrix }}" +matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_hostname }}`)" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" diff --git a/roles/custom/matrix-nginx-proxy/tasks/validate_config.yml b/roles/custom/matrix-nginx-proxy/tasks/validate_config.yml index f7d18c9e..8d63876a 100644 --- a/roles/custom/matrix-nginx-proxy/tasks/validate_config.yml +++ b/roles/custom/matrix-nginx-proxy/tasks/validate_config.yml @@ -14,6 +14,8 @@ - {'old': 'matrix_nginx_proxy_proxy_riot_enabled', 'new': 'matrix_nginx_proxy_proxy_element_enabled'} - {'old': 'matrix_ssl_lets_encrypt_renew_cron_time_definition', 'new': ''} - {'old': 'matrix_nginx_proxy_reload_cron_time_definition', 'new': ''} + - {'old': 'matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule', 'new': ''} + - {'old': 'matrix_nginx_proxy_container_labels_traefik_proxy_matrix_hostname', 'new': ''} - name: Fail on unknown matrix_ssl_retrieval_method ansible.builtin.fail: diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index b4c68471..0aff6a05 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -20,7 +20,7 @@ traefik.http.routers.matrix-nginx-proxy-base-domain.entrypoints={{ matrix_nginx_ {% if matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled %} # Matrix Client -traefik.http.routers.matrix-nginx-proxy-matrix-client.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule }} +traefik.http.routers.matrix-nginx-proxy-matrix-client.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_client_rule }} traefik.http.routers.matrix-nginx-proxy-matrix-client.service=matrix-nginx-proxy-web traefik.http.routers.matrix-nginx-proxy-matrix-client.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls | to_json }} {% if matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls %} @@ -29,7 +29,7 @@ traefik.http.routers.matrix-nginx-proxy-matrix-client.tls.certResolver={{ matrix traefik.http.routers.matrix-nginx-proxy-matrix-client.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} # Matrix Federation -traefik.http.routers.matrix-nginx-proxy-matrix-federation.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_rule }} +traefik.http.routers.matrix-nginx-proxy-matrix-federation.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_rule }} traefik.http.routers.matrix-nginx-proxy-matrix-federation.service=matrix-nginx-proxy-federation traefik.http.routers.matrix-nginx-proxy-matrix-federation.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls | to_json }} {% if matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls %} From 91660f7433867b0ea4d1165c53d76b05e4d4387a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 17:45:28 +0200 Subject: [PATCH 256/382] Improve wordng in examples/nginx/README.md --- examples/nginx/README.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/examples/nginx/README.md b/examples/nginx/README.md index 038b060e..f9be21fd 100644 --- a/examples/nginx/README.md +++ b/examples/nginx/README.md @@ -1,11 +1,17 @@ -# Nginx reverse-proxy fronting playbook's Traefik +# Nginx reverse-proxy fronting the playbook's integrated Traefik reverse-proxy -This directory contains a sample config that show you how to do reverse-proxying using Nginx and the playbook's internal traefik container. +This directory contains a sample config that shows you how use the [nginx](https://nginx.org/) webserver to front the integrated [Traefik](https://traefik.io/) reverse-proxy webserver with another reverse-proxy. -This is for when you wish to front the playbook's integrated traefik container with a self-managed Nginx reverse-proxy running on the same server. -See the [Using your own webserver, instead of this playbook's nginx proxy & Fronting the integrated reverse-proxy webserver with another reverse-proxy](../../docs/configuring-playbook-own-webserver.md#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy) documentation page and follow the instructions for the playbook's configuration (`inventory/host_vars/matrix./vars.yml`). -**NOTE**: -- that this also disables SSL certificate retrieval, which then has to be done manually (e.g. by using certbot and setting the appropriate path as found in [the example nginx configuration file](./matrix.conf)). For the example nginx config one certificate is used that contains all the used subdomains. -- that [the example nginx configuration file](./matrix.conf) has to be adapted to whatever services you are using. For example, remove element.domain.com from the `server_name` list if you don't use Element web client or add dimension.domain.com to it if you do use Dimension. -- that this is just an example and may not be entirely accurate. It may also not cover other use cases (enabling various services or bridges requires additional reverse-proxying configuration). +## Prerequisite configuration + +To get started, first follow the [front the integrated reverse-proxy webserver with another reverse-proxy](../../docs/configuring-playbook-own-webserver.md#fronting-the-integrated-reverse-proxy-webserver-with-another-reverse-proxy) instructions and update your playbook's configuration (`inventory/host_vars/matrix./vars.yml`). + + +## Using the nginx configuration + +Copy the [matrix.conf](matrix.conf) file to your nginx server's filesystem, modify it to your needs and include it your nginx configuration (e.g. `include /path/to/matrix.conf;`). + +This configuration **disables SSL certificate retrieval**, so you will **need to obtain SSL certificates manually** (e.g. by using [certbot](https://certbot.eff.org/)) and set the appropriate path in `matrix.conf`. In the example nginx configuration, a single certificate is used for all subdomains (`matrix.DOMAIN`, `element.DOMAIN`, etc.). For your setup, may wish to change this and use separate `server` blocks and separate certificate files for each host. + +Also note that your copy of the `matrix.conf` file has to be adapted to whatever services you are using. For example, remove `element.domain.com` from the `server_name` list if you don't use [Element](../../docs/configuring-playbook-client-element.md) web client or add `dimension.domain.com` to it if you do use the [Dimension](../../docs/configuring-playbook-dimension.md) integration manager. From faa074f7af4ca4b5fa86122a38a303ab50e3cf57 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 17:47:19 +0200 Subject: [PATCH 257/382] Improve wording --- docs/configuring-playbook-own-webserver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index cae1486e..81b154e2 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -143,7 +143,7 @@ devture_traefik_additional_entrypoints_auto: config: {} ``` -For an example where the playbook's traefik is fronted by Nginx running on the same server, see [Nginx reverse-proxy fronting playbook's Traefik](../examples/nginx/README.md). +For an example where the playbook's Traefik reverse-proxy is fronted by [Nginx](https://nginx.org/) running on the same server, see [Nginx reverse-proxy fronting the playbook's Traefik](../examples/nginx/README.md). (Deprecated) **For `matrix-nginx-proxy`** fronted by another reverse-proxy, you would need some configuration like this: From 73f09d4bb0aa4bf6006267f3af62cf1c1ea255d7 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 17:48:35 +0200 Subject: [PATCH 258/382] Remove fronting matrix-nginx-proxy example We don't need this anymore. No one should be using matrix-nginx-proxy anymore. --- docs/configuring-playbook-own-webserver.md | 32 ++-------------------- 1 file changed, 3 insertions(+), 29 deletions(-) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 81b154e2..12c969d2 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -123,7 +123,7 @@ You can disable such behavior and make the integrated reverse-proxy webserver on This is the recommended way for using another reverse-proxy, because the integrated one would act as a black box and wire all Matrix services correctly. You would only need to reverse-proxy a few individual domains and ports over to it. -**For Traefik** fronted by another reverse-proxy, you would need some configuration like this: +To front Traefik with another reverse-proxy, you would need some configuration like this: ```yaml matrix_playbook_reverse_proxy_type: playbook-managed-traefik @@ -143,36 +143,10 @@ devture_traefik_additional_entrypoints_auto: config: {} ``` -For an example where the playbook's Traefik reverse-proxy is fronted by [Nginx](https://nginx.org/) running on the same server, see [Nginx reverse-proxy fronting the playbook's Traefik](../examples/nginx/README.md). - -(Deprecated) **For `matrix-nginx-proxy`** fronted by another reverse-proxy, you would need some configuration like this: - -```yaml -matrix_playbook_reverse_proxy_type: playbook-managed-nginx - -# Ensure that public urls use https -matrix_playbook_ssl_enabled: true - -# Disable SSL certificate retrieval -matrix_ssl_retrieval_method: none - -# Given that we won't be obtaining SSL certificates here, disable HTTPS -matrix_nginx_proxy_https_enabled: false - -# Do not listen for HTTP on port 80 globally (default), listen on the loopback interface. -# If you'd like, you can make it use the local network as well and reverse-proxy from another local machine. -matrix_nginx_proxy_container_http_host_bind_port: '127.0.0.1:81' - -# Likewise, expose the Matrix Federation port on the loopback interface. -# Since `matrix_nginx_proxy_https_enabled` is set to `false`, this federation port will serve HTTP traffic. -# If you'd like, you can make it use the local network as well and reverse-proxy from another local machine. -# -# You'd most likely need to expose it publicly on port 8448 (8449 was chosen for the local port to prevent overlap). -matrix_nginx_proxy_container_federation_host_bind_port: '127.0.0.1:8449' -``` - If you'll be fronting with a reverse-proxy that lives on another machine (not on the same one as Matrix), you need to replace `127.0.0.1` in the above configurations with `0.0.0.0` or another network interface. +For an example where the playbook's Traefik reverse-proxy is fronted by [Nginx](https://nginx.org/) running on the same server, see [Nginx reverse-proxy fronting the playbook's Traefik](../examples/nginx/README.md). + ### Using no reverse-proxy on the Matrix side at all From bb9053687486bc320b31fb735b3ab6abab7673d5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 17:57:33 +0200 Subject: [PATCH 259/382] Fix variable name typos in docs/configuring-playbook-postgres-backup.md Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2524 --- docs/configuring-playbook-postgres-backup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configuring-playbook-postgres-backup.md b/docs/configuring-playbook-postgres-backup.md index 2510ecc1..9ea3c60e 100644 --- a/docs/configuring-playbook-postgres-backup.md +++ b/docs/configuring-playbook-postgres-backup.md @@ -23,8 +23,8 @@ Refer to the table below for additional configuration variables and their defaul |`devture_postgres_backup_keep_days`|`7`|Number of daily backups to keep| |`devture_postgres_backup_keep_weeks`|`4`|Number of weekly backups to keep| |`devture_postgres_backup_keep_months`|`12`|Number of monthly backups to keep| -|`devture_postgres_base_path` | `"{{ matrix_base_data_path }}/postgres-backup"` | Base path for postgres-backup. Also see `devture_postgres_data_path` | -|`devture_postgres_data_path` | `"{{ devture_postgres_base_path }}/data"` | Storage path for postgres-backup database backups | +|`devture_postgres_backup_base_path` | `"{{ matrix_base_data_path }}/postgres-backup"` | Base path for postgres-backup. Also see `devture_postgres_backup_data_path` | +|`devture_postgres_backup_data_path` | `"{{ devture_postgres_backup_base_path }}/data"` | Storage path for postgres-backup database backups | ## Installing From 6fe739a2cb790add5a307bd35255c13edbc8096d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 27 Feb 2023 18:32:22 +0200 Subject: [PATCH 260/382] Remove unnecessary variable --- roles/custom/matrix-nginx-proxy/defaults/main.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 6565e1b9..27978cae 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -55,7 +55,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_tls: "{{ matrix_ng matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_hostname }}`)" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_matrix_hostname: "{{ matrix_server_fqn_matrix }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_client_hostname: "{{ matrix_server_fqn_matrix }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_client_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_client_hostname }}`)" From 0aede060f30a35f0c789701dc96b04cb9eff6ae8 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 28 Feb 2023 13:18:21 +0000 Subject: [PATCH 261/382] Update grafana 9.3.6 -> 9.4.1 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index c882aee9..e638955e 100644 --- a/requirements.yml +++ b/requirements.yml @@ -40,7 +40,7 @@ version: v1.2.3-1.7.7-1 - src: git+https://gitlab.com/etke.cc/roles/grafana.git - version: v9.3.6-2 + version: v9.4.1-0 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git version: v2.1.0-0 From 0fe1d1f9ece89025ee5952a4730e03e83fcd2654 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 28 Feb 2023 13:18:59 +0000 Subject: [PATCH 262/382] Update element 1.11.23 -> 1.11.24 --- roles/custom/matrix-client-element/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 60b52d60..8e24ca66 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -10,7 +10,7 @@ matrix_client_element_container_image_self_build_repo: "https://github.com/vecto # - https://github.com/vector-im/element-web/issues/19544 matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}" -matrix_client_element_version: v1.11.23 +matrix_client_element_version: v1.11.24 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" From 32e700f0767f8893fe2a0f5fd0b65fdd4b96a7b6 Mon Sep 17 00:00:00 2001 From: plui29989 Date: Tue, 28 Feb 2023 21:07:16 +0100 Subject: [PATCH 263/382] Add doc for self-signed certificates --- docs/configuring-playbook-ssl-certificates.md | 43 ++++++++++++++++++- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index 60d5b135..a2fab7c3 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -29,6 +29,45 @@ devture_traefik_config_entrypoint_web_secure_enabled: false ## Using self-signed SSL certificates -Using self-signed certificates with Traefik is a somewhat involved processes, where you need to manually mount the files into the container and adjust the "static" configuration to refer to them. +To use self-signed SSL certificates, you need to disable the certResolvers and the traefik-certs-dumper tool. +You also need to override the providers.file setting in the Traefik configs. +Create a file 'certificates.yml' in /devture-traefik/config/ with the following content: -Feel free to research this approach on your own and improve this guide! +```yaml +tls: + certificates: + - certFile: /ssl/cert.pem + keyFile: /ssl/privkey.pem + stores: + default: + defaultCertificate: + certFile: /ssl/cert.pem + keyFile: /ssl/privkey.pem +``` + +Place the key and your certificate in /devture-traefik/ssl/ +You can use the matrix-aux role for this: + +```yaml +matrix_aux_file_definitions: + - dest: /devture-traefik/ssl/privkey.pem + src: /path/to/privkey.pem + - dest: /devture-traefik/ssl/cert.pem + src: /path/to/cert.pem + - dest: /devture-traefik/config/certificates.yml + src: /path/to/certificates.yml +``` + +Then add the following to your vars.yml: + +```yaml +devture_traefik_config_certificatesResolvers_acme_enabled: false +devture_traefik_certResolver_primary: '' +devture_traefik_ssl_dir_enabled: true +devture_traefik_configuration_extension_yaml: | + providers: + file: + filename: /config/certificates.yml + watch: true +matrix_playbook_traefik_certs_dumper_role_enabled: false +``` From 4b17a1e73a5a3560f6b2bb25ec8ac6ac26c52924 Mon Sep 17 00:00:00 2001 From: plui29989 Date: Tue, 28 Feb 2023 21:09:37 +0100 Subject: [PATCH 264/382] formatting --- docs/configuring-playbook-ssl-certificates.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index a2fab7c3..859885dd 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -31,6 +31,7 @@ devture_traefik_config_entrypoint_web_secure_enabled: false To use self-signed SSL certificates, you need to disable the certResolvers and the traefik-certs-dumper tool. You also need to override the providers.file setting in the Traefik configs. + Create a file 'certificates.yml' in /devture-traefik/config/ with the following content: ```yaml @@ -46,6 +47,7 @@ tls: ``` Place the key and your certificate in /devture-traefik/ssl/ + You can use the matrix-aux role for this: ```yaml From c3900aaf1b7c755519857f315da56ac3e304bbf5 Mon Sep 17 00:00:00 2001 From: array-in-a-matrix Date: Tue, 28 Feb 2023 18:06:46 -0500 Subject: [PATCH 265/382] remove devture services --- roles/custom/matrix-base/templates/bin/remove-all.j2 | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/roles/custom/matrix-base/templates/bin/remove-all.j2 b/roles/custom/matrix-base/templates/bin/remove-all.j2 index 5dd862ae..46e1f757 100644 --- a/roles/custom/matrix-base/templates/bin/remove-all.j2 +++ b/roles/custom/matrix-base/templates/bin/remove-all.j2 @@ -21,6 +21,13 @@ else rm -f {{ devture_systemd_docker_base_systemd_path }}/$s done + echo "Stop and remove devture services" + + for s in $(find {{ devture_systemd_docker_base_systemd_path }}/ -type f -name "devture-*" -printf "%f\n"); do + systemctl disable --now $s + rm -f {{ devture_systemd_docker_base_systemd_path }}/$s + done + systemctl daemon-reload echo "Remove unused Docker images and resources" From f7149103e457eaf1067824360ecd95c59ec69c6f Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 09:29:14 +0200 Subject: [PATCH 266/382] Remove matrix_playbook_traefik_certs_dumper_role_enabled in favor of just devture_traefik_certs_dumper_enabled We don't need these 2 roughly-the-same settings related to the traefik-certs-dumper role. For Traefik, it makes sense, because it's a component used by the various related playbooks and they could step onto each other's toes if the role is enabled, but Traefik is disabled (in that case, uninstall tasks will run). As for Traefik certs dumper, the other related playbooks don't have it, so there's no conflict. Even if they used it, each one would use its own instance (different `devture_traefik_certs_dumper_identifier`), so there wouldn't be a conflict and uninstall tasks can run without any danger. --- group_vars/matrix_servers | 16 ++++++---------- playbooks/matrix.yml | 3 +-- .../tasks/validate_config.yml | 1 + 3 files changed, 8 insertions(+), 12 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 4827a3c2..4195aa52 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -38,12 +38,10 @@ matrix_playbook_traefik_role_enabled: "{{ matrix_playbook_reverse_proxy_type != # installed in another way. matrix_playbook_traefik_labels_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" -matrix_playbook_traefik_certs_dumper_role_enabled: "{{ (matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' and devture_traefik_config_entrypoint_web_secure_enabled) or matrix_playbook_reverse_proxy_type == 'other-traefik-container' }}" - # Controls the additional network that reverse-proxyable services will be connected to. matrix_playbook_reverse_proxyable_services_additional_network: "{{ devture_traefik_container_network if devture_traefik_enabled else '' }}" -matrix_playbook_ssl_retrieval_method: "{{ 'lets-encrypt' if matrix_playbook_traefik_certs_dumper_role_enabled else matrix_ssl_retrieval_method }}" +matrix_playbook_ssl_retrieval_method: "{{ 'lets-encrypt' if devture_traefik_certs_dumper_enabled else matrix_ssl_retrieval_method }}" matrix_playbook_ssl_enabled: "{{ matrix_playbook_ssl_retrieval_method in ['lets-encrypt', 'self-signed', 'manually-managed'] }}" @@ -360,7 +358,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'devture-traefik.service', 'priority': 3000, 'groups': ['matrix', 'traefik', 'reverse-proxies']}] if devture_traefik_enabled else []) + - ([{'name': (devture_traefik_certs_dumper_identifier + '.service'), 'priority': 3500, 'groups': ['matrix', 'traefik-certs-dumper']}] if matrix_playbook_traefik_certs_dumper_role_enabled else []) + ([{'name': (devture_traefik_certs_dumper_identifier + '.service'), 'priority': 3500, 'groups': ['matrix', 'traefik-certs-dumper']}] if devture_traefik_certs_dumper_enabled else []) }} ######################################################################## @@ -1789,7 +1787,7 @@ matrix_bot_postmoogle_systemd_required_services_list: | + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + - (matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services | trim | split(' ') if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled else []) + (matrix_playbook_bot_postmoogle_traefik_certs_dumper_waiter_services | trim | split(' ') if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and devture_traefik_certs_dumper_enabled else []) }} # Postgres is the default, except if not using internal Postgres server @@ -2131,7 +2129,7 @@ matrix_coturn_container_additional_volumes: | 'dst': '/privatekey.key', 'options': 'ro', }, - ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled and matrix_coturn_tls_enabled else [] + ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and devture_traefik_certs_dumper_enabled and matrix_coturn_tls_enabled else [] ) }} @@ -2139,7 +2137,7 @@ matrix_coturn_systemd_required_services_list: | {{ ['docker.service'] + - ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and matrix_playbook_traefik_certs_dumper_role_enabled and matrix_coturn_tls_enabled else []) + ([devture_traefik_certs_dumper_identifier + '-wait-for-domain@' + matrix_server_fqn_matrix + '.service'] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and devture_traefik_certs_dumper_enabled and matrix_coturn_tls_enabled else []) }} ###################################################################### @@ -3807,9 +3805,7 @@ devture_traefik_additional_domains_to_obtain_certificates_for: "{{ matrix_ssl_ad # # ######################################################################## -# To completely disable the Traefik certs dumper role from running, use `matrix_playbook_traefik_certs_dumper_role_enabled: false`. -# See the comment there for more details about why we have both `devture_traefik_certs_dumper_enabled` and `matrix_playbook_traefik_certs_dumper_role_enabled`. -devture_traefik_certs_dumper_enabled: "{{ matrix_playbook_traefik_certs_dumper_role_enabled }}" +devture_traefik_certs_dumper_enabled: "{{ (matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' and devture_traefik_config_entrypoint_web_secure_enabled) or matrix_playbook_reverse_proxy_type == 'other-traefik-container' }}" devture_traefik_certs_dumper_identifier: matrix-traefik-certs-dumper diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index a425cd88..6a8987bf 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -121,8 +121,7 @@ - when: matrix_playbook_traefik_role_enabled | bool role: galaxy/com.devture.ansible.role.traefik - - when: matrix_playbook_traefik_certs_dumper_role_enabled | bool - role: galaxy/com.devture.ansible.role.traefik_certs_dumper + - role: galaxy/com.devture.ansible.role.traefik_certs_dumper - when: devture_systemd_service_manager_enabled | bool role: galaxy/com.devture.ansible.role.systemd_service_manager diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 7afd2154..2241d502 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -36,6 +36,7 @@ - {'old': 'matrix_systemd_services_autostart_enabled', 'new': 'devture_systemd_service_manager_services_autostart_enabled'} - {'old': 'matrix_prometheus_node_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled'} - {'old': 'matrix_prometheus_postgres_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled'} + - {'old': 'matrix_playbook_traefik_certs_dumper_role_enabled', 'new': 'devture_traefik_certs_dumper_enabled'} - name: (Deprecation) Catch and report matrix_postgres variables ansible.builtin.fail: From b0845984b3ea9e73e78f020ae3690db74363da49 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 09:44:54 +0200 Subject: [PATCH 267/382] Only enable Traefik certs dumper if the ACME certificate resolver for Traefik is enabled If someone disables ACME, then they're using their own certificates somehow. There's nothing to dump from an `acme.json` file. --- group_vars/matrix_servers | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 4195aa52..6e40aba7 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3805,7 +3805,7 @@ devture_traefik_additional_domains_to_obtain_certificates_for: "{{ matrix_ssl_ad # # ######################################################################## -devture_traefik_certs_dumper_enabled: "{{ (matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' and devture_traefik_config_entrypoint_web_secure_enabled) or matrix_playbook_reverse_proxy_type == 'other-traefik-container' }}" +devture_traefik_certs_dumper_enabled: "{{ (matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' and devture_traefik_config_entrypoint_web_secure_enabled and devture_traefik_config_certificatesResolvers_acme_enabled) or matrix_playbook_reverse_proxy_type == 'other-traefik-container' }}" devture_traefik_certs_dumper_identifier: matrix-traefik-certs-dumper From 7331d314c422ec9b113ea3d09135355d89610824 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 09:45:54 +0200 Subject: [PATCH 268/382] Improve wording --- docs/configuring-playbook-ssl-certificates.md | 74 +++++++++++-------- 1 file changed, 42 insertions(+), 32 deletions(-) diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index 859885dd..bde425bd 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -29,47 +29,57 @@ devture_traefik_config_entrypoint_web_secure_enabled: false ## Using self-signed SSL certificates -To use self-signed SSL certificates, you need to disable the certResolvers and the traefik-certs-dumper tool. -You also need to override the providers.file setting in the Traefik configs. +To use self-signed SSL certificates, you need to: -Create a file 'certificates.yml' in /devture-traefik/config/ with the following content: - -```yaml -tls: - certificates: - - certFile: /ssl/cert.pem - keyFile: /ssl/privkey.pem - stores: - default: - defaultCertificate: - certFile: /ssl/cert.pem - keyFile: /ssl/privkey.pem -``` - -Place the key and your certificate in /devture-traefik/ssl/ - -You can use the matrix-aux role for this: - -```yaml -matrix_aux_file_definitions: - - dest: /devture-traefik/ssl/privkey.pem - src: /path/to/privkey.pem - - dest: /devture-traefik/ssl/cert.pem - src: /path/to/cert.pem - - dest: /devture-traefik/config/certificates.yml - src: /path/to/certificates.yml -``` - -Then add the following to your vars.yml: +- disable `certResolvers` in Traefik, so it won't attempt to retrieve SSL certificates using the default certificate resolver (using [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment) / [Let's Encrypt](https://letsencrypt.org/)) +- put a custom Traefik configuration file on the server, with the help of this Ansible playbook (via the `matrix-aux` role) or manually +- register your custom configuration file with Traefik, by adding an extra provider of type [file](https://doc.traefik.io/traefik/providers/file/) +- put the SSL files on the server, with the help of this Ansible playbook (via the `matrix-aux` role) or manually ```yaml +# Disable ACME / Let's Encrypt support devture_traefik_config_certificatesResolvers_acme_enabled: false + +# Unset the default certificate resolver devture_traefik_certResolver_primary: '' + +# Keep the SSL directory normally used for ACME / Let's Encrypt certificates. +# We need to explicitly enable this, because disabling ACME support (above) automatically disables it otherwise. devture_traefik_ssl_dir_enabled: true + +# Tell Traefik to load our custom configuration file (certificates.yml). +# The file is created below. See `matrix_aux_file_definitions`. +# The `/config/..` path is an in-container path, not a path on the host. Do not change it! devture_traefik_configuration_extension_yaml: | providers: file: filename: /config/certificates.yml watch: true -matrix_playbook_traefik_certs_dumper_role_enabled: false + +# Use the matrix-aux role to create our custom files on the server. +# If you'd like to do this manually, you remove this `matrix_aux_file_definitions` variable. +matrix_aux_file_definitions: + # Create the privkey.pem file on the server by + # uploading a file from the computer where Ansible is running. + - dest: "{{ devture_traefik_ssl_dir_path }}/privkey.pem" + src: /path/on/your/Ansible/computer/to/privkey.pem + + # Create the cert.pem file on the server + # uploading a file from the computer where Ansible is running. + - dest: "{{ devture_traefik_ssl_dir_path }}/cert.pem" + src: /path/on/your/Ansible/computer/to/cert.pem + + # Create the custom Traefik configuration. + # The `/ssl/..` paths below are in-container paths, not paths on the host. Do not change them! + - dest: "{{ devture_traefik_config_dir_path }}/certificates.yml" + content: | + tls: + certificates: + - certFile: /ssl/cert.pem + keyFile: /ssl/privkey.pem + stores: + default: + defaultCertificate: + certFile: /ssl/cert.pem + keyFile: /ssl/privkey.pem ``` From ba2a3caaf383fa7b576d156acfd8c87584ff850b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 09:54:23 +0200 Subject: [PATCH 269/382] Minor improvements to docs/configuring-playbook-ssl-certificates.md --- docs/configuring-playbook-ssl-certificates.md | 37 +++++++++++++------ 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index bde425bd..13d5f609 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -29,27 +29,32 @@ devture_traefik_config_entrypoint_web_secure_enabled: false ## Using self-signed SSL certificates -To use self-signed SSL certificates, you need to: +If you'd like to use your own SSL certificates, instead of the default (SSL certificates obtained automatically via [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment) from [Let's Encrypt](https://letsencrypt.org/)): -- disable `certResolvers` in Traefik, so it won't attempt to retrieve SSL certificates using the default certificate resolver (using [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment) / [Let's Encrypt](https://letsencrypt.org/)) +- generate your self-signed certificate files +- follow the [Using your own SSL certificates](#using-your-own-ssl-certificates) documentation below + + +## Using your own SSL certificates + +To use your own SSL certificates with Traefik, you need to: + +- disable [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment) / [Let's Encrypt](https://letsencrypt.org/) support - put a custom Traefik configuration file on the server, with the help of this Ansible playbook (via the `matrix-aux` role) or manually - register your custom configuration file with Traefik, by adding an extra provider of type [file](https://doc.traefik.io/traefik/providers/file/) - put the SSL files on the server, with the help of this Ansible playbook (via the `matrix-aux` role) or manually ```yaml -# Disable ACME / Let's Encrypt support +# Disable ACME / Let's Encrypt support. devture_traefik_config_certificatesResolvers_acme_enabled: false -# Unset the default certificate resolver -devture_traefik_certResolver_primary: '' - -# Keep the SSL directory normally used for ACME / Let's Encrypt certificates. -# We need to explicitly enable this, because disabling ACME support (above) automatically disables it otherwise. +# Disabling ACME support (above) automatically disables the creation of the SSL directory. +# Force-enable it here, because we'll add our certificate files there. devture_traefik_ssl_dir_enabled: true # Tell Traefik to load our custom configuration file (certificates.yml). -# The file is created below. See `matrix_aux_file_definitions`. -# The `/config/..` path is an in-container path, not a path on the host. Do not change it! +# The file is created below, in `matrix_aux_file_definitions`. +# The `/config/..` path is an in-container path, not a path on the host (like `/devture-traefik/config`). Do not change it! devture_traefik_configuration_extension_yaml: | providers: file: @@ -63,14 +68,24 @@ matrix_aux_file_definitions: # uploading a file from the computer where Ansible is running. - dest: "{{ devture_traefik_ssl_dir_path }}/privkey.pem" src: /path/on/your/Ansible/computer/to/privkey.pem + # Alternatively, comment out `src` above and uncomment the lines below to provide the certificate content inline. + # Note the indentation level. + # content: | + # FILE CONTENT + # HERE # Create the cert.pem file on the server # uploading a file from the computer where Ansible is running. - dest: "{{ devture_traefik_ssl_dir_path }}/cert.pem" src: /path/on/your/Ansible/computer/to/cert.pem + # Alternatively, comment out `src` above and uncomment the lines below to provide the certificate content inline. + # Note the indentation level. + # content: | + # FILE CONTENT + # HERE # Create the custom Traefik configuration. - # The `/ssl/..` paths below are in-container paths, not paths on the host. Do not change them! + # The `/ssl/..` paths below are in-container paths, not paths on the host (/`devture-traefik/ssl/..`). Do not change them! - dest: "{{ devture_traefik_config_dir_path }}/certificates.yml" content: | tls: From 468bed653e06ad632cdbdaf9c6c1cb66a44b0eee Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 10:30:09 +0200 Subject: [PATCH 270/382] Upgrade Redis (v7.0.7-0 -> v7.0.9-0) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index e638955e..fc39323a 100644 --- a/requirements.yml +++ b/requirements.yml @@ -46,7 +46,7 @@ version: v2.1.0-0 - src: git+https://gitlab.com/etke.cc/roles/redis.git - version: v7.0.7-0 + version: v7.0.9-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: acc49a8059206743f1921f94891ebc177bf79bd2 From d6371a07816bf0861d653c70188049b21eb5175b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 10:32:49 +0200 Subject: [PATCH 271/382] Upgrade ddclient (v3.10.0-ls110 -> v3.10.0-ls111) --- roles/custom/matrix-dynamic-dns/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-dynamic-dns/defaults/main.yml b/roles/custom/matrix-dynamic-dns/defaults/main.yml index 4ec034a9..08cf023c 100644 --- a/roles/custom/matrix-dynamic-dns/defaults/main.yml +++ b/roles/custom/matrix-dynamic-dns/defaults/main.yml @@ -7,7 +7,7 @@ matrix_dynamic_dns_enabled: true # The dynamic dns daemon interval matrix_dynamic_dns_daemon_interval: '300' -matrix_dynamic_dns_version: v3.10.0-ls110 +matrix_dynamic_dns_version: v3.10.0-ls111 # The docker container to use when in mode matrix_dynamic_dns_docker_image: "{{ matrix_dynamic_dns_docker_image_name_prefix }}linuxserver/ddclient:{{ matrix_dynamic_dns_version }}" From 8db2c0498a20aaf74a59a3f8bfcdd2eef7f30c02 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 11:07:25 +0200 Subject: [PATCH 272/382] Upgrade Synapse (v1.77.0 -> v1.78.0) --- roles/custom/matrix-synapse/defaults/main.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index dd69facd..41ad0bc4 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -4,6 +4,8 @@ matrix_synapse_enabled: true +matrix_synapse_version: v1.78.0 + matrix_synapse_username: '' matrix_synapse_uid: '' matrix_synapse_gid: '' @@ -69,7 +71,6 @@ matrix_synapse_container_image_customizations_dockerfile_body_custom: '' matrix_synapse_docker_image: "{{ matrix_synapse_docker_image_name_prefix }}matrixdotorg/synapse:{{ matrix_synapse_docker_image_tag }}" matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_container_image_self_build else matrix_container_global_registry_prefix }}" -matrix_synapse_version: v1.77.0 matrix_synapse_docker_image_tag: "{{ matrix_synapse_version }}" matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}" From a4b401c4da53ada67f25dd131144bce880ac5593 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 12:31:46 +0200 Subject: [PATCH 273/382] Upgrade com.devture.ansible.role.traefik and improve front-Traefik-with-another-proxy docs Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2532 --- docs/configuring-playbook-own-webserver.md | 19 +++++++++++++++++-- requirements.yml | 2 +- 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 12c969d2..affb565f 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -134,17 +134,32 @@ matrix_playbook_ssl_enabled: true # Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval devture_traefik_config_entrypoint_web_secure_enabled: false +# If your reverse-proxy runs on another machine, consider using `0.0.0.0:81`, just `81` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:81` devture_traefik_container_web_host_bind_port: '127.0.0.1:81' +# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from +# a reverse-proxy running on the local machine is safe enough. +devture_traefik_config_entrypoint_web_forwardedHeaders_insecure: true + +# Or, if you're publishing the port (`devture_traefik_container_web_host_bind_port` above) to a public network interfaces: +# - remove the `devture_traefik_config_entrypoint_web_forwardedHeaders_insecure` variable definition above +# - uncomment and adjust the line below +# devture_traefik_config_entrypoint_web_forwardedHeaders_trustedIPs: ['IP-ADDRESS-OF-YOUR-REVERSE-PROXY'] + +# Likewise (to `devture_traefik_container_web_host_bind_port` above), +# if your reverse-proxy runs on another machine, consider changing the `host_bind_port` setting below. devture_traefik_additional_entrypoints_auto: - name: matrix-federation port: 8449 host_bind_port: '127.0.0.1:8449' config: {} + # If your reverse-proxy runs on another machine, remove the config above and use this config instead: + # config: + # forwardedHeaders: + # insecure: true + # # trustedIPs: ['IP-ADDRESS-OF-YOUR-REVERSE-PROXY'] ``` -If you'll be fronting with a reverse-proxy that lives on another machine (not on the same one as Matrix), you need to replace `127.0.0.1` in the above configurations with `0.0.0.0` or another network interface. - For an example where the playbook's Traefik reverse-proxy is fronted by [Nginx](https://nginx.org/) running on the same server, see [Nginx reverse-proxy fronting the playbook's Traefik](../examples/nginx/README.md). diff --git a/requirements.yml b/requirements.yml index fc39323a..1c3b8e76 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.9-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: acc49a8059206743f1921f94891ebc177bf79bd2 + version: ed9f59753468556b363f90798be23c600d16ef5f - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: abeecc09f6fd165eff3cef5d6dd7696b9ca4ec27 From adc18251a9e9a34ba51b49e9c6eb91dca01a4f22 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 12:47:55 +0200 Subject: [PATCH 274/382] Upgrade com.devture.ansible.role.traefik (2.9.6 -> 2.9.8) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 1c3b8e76..410dc446 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.9-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: ed9f59753468556b363f90798be23c600d16ef5f + version: 4ec9187017cb7832f521fc273fabd0a873ca2736 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: abeecc09f6fd165eff3cef5d6dd7696b9ca4ec27 From 4cd9e65d6dc47166b8584870ac88f84ebedb439d Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Wed, 1 Mar 2023 20:09:54 +0000 Subject: [PATCH 275/382] update ntfy 2.1.0 -> 2.1.1 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 410dc446..c8d6ed7d 100644 --- a/requirements.yml +++ b/requirements.yml @@ -43,7 +43,7 @@ version: v9.4.1-0 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git - version: v2.1.0-0 + version: v2.1.1-0 - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.9-0 From 227d9d820900d15fd2a3dbf06283c805b721928d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 1 Mar 2023 23:14:40 +0200 Subject: [PATCH 276/382] Fix variable name typo --- roles/custom/matrix-bot-maubot/tasks/setup_install.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-maubot/tasks/setup_install.yml b/roles/custom/matrix-bot-maubot/tasks/setup_install.yml index 97f189c4..60b87861 100644 --- a/roles/custom/matrix-bot-maubot/tasks/setup_install.yml +++ b/roles/custom/matrix-bot-maubot/tasks/setup_install.yml @@ -53,7 +53,7 @@ name: "{{ matrix_bot_maubot_docker_image }}" source: build force_source: "{{ matrix_bot_maubot_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_maubot_git_pull_results.changed }}" build: dockerfile: Dockerfile path: "{{ matrix_bot_maubot_docker_src_files_path }}" From 4ccd3f79de54d3c86a1cebb0ea4ec8c33ddde8d4 Mon Sep 17 00:00:00 2001 From: Catalan Lover Date: Thu, 2 Mar 2023 03:58:03 +0100 Subject: [PATCH 277/382] Fix Draupnir config Config was accidently changed when importing upstream changes. --- .../matrix-bot-draupnir/templates/production.yaml.j2 | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 index 2dbff886..95acbd35 100644 --- a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 +++ b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 @@ -36,7 +36,7 @@ autojoinOnlyIfManager: true # If `autojoinOnlyIfManager` is false, only the members in this space can invite # the bot to new rooms. -acceptInvitesFromSpace: "!example:example.org" +#acceptInvitesFromSpace: "!example:example.org" # Whether Draupnir should report ignored invites to the management room (if autojoinOnlyIfManager is true). recordIgnoredInvites: false @@ -49,11 +49,11 @@ recordIgnoredInvites: false # # Note: By default, Draupnir is fairly verbose - expect a lot of messages in this room. # (see verboseLogging to adjust this a bit.) -managementRoom: "#moderators:example.org" +managementRoom: "{{ matrix_bot_draupnir_management_room }}" # Whether Draupnir should log a lot more messages in the room, # mainly involves "all-OK" messages, and debugging messages for when Draupnir checks bans in a room. -verboseLogging: true +verboseLogging: false # The log level of terminal (or container) output, # can be one of DEBUG, INFO, WARN and ERROR, in increasing order of importance and severity. @@ -130,7 +130,7 @@ admin: # (with enough permissions) to "make" a user an admin. # # This only works if a local user with enough admin permissions is present in the room. - enableMakeRoomAdminCommand: true + enableMakeRoomAdminCommand: false # Misc options for command handling and commands commands: @@ -146,6 +146,7 @@ commands: # Any additional bot prefixes that Draupnir will listen to. i.e. adding `mod` will allow `!mod help`. additionalPrefixes: - "draupnir-bot" + - "draupnir_bot" - "draupnir" # Whether or not commands with a wildcard (*) will require an additional `--force` argument From 4f74d21ba8d53f6b9c7651ddf3e6568f49bd705b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 2 Mar 2023 10:31:23 +0200 Subject: [PATCH 278/382] Upgrade devture/ansible (2.13.6-r0-1 -> 2.13.6-r0-2) The new version includes the `passlib` Python module. --- docs/ansible.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ansible.md b/docs/ansible.md index 60be6164..dcda338a 100644 --- a/docs/ansible.md +++ b/docs/ansible.md @@ -65,7 +65,7 @@ docker run -it --rm \ -w /work \ -v `pwd`:/work \ --entrypoint=/bin/sh \ -docker.io/devture/ansible:2.13.6-r0-1 +docker.io/devture/ansible:2.13.6-r0-2 ``` Once you execute the above command, you'll be dropped into a `/work` directory inside a Docker container. From ae76db4d77d3ee52e69c7d254122747f31a010da Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 2 Mar 2023 16:06:01 +0200 Subject: [PATCH 279/382] Upgrade com.devture.ansible.role.traefik_certs_dumper for wait time increase (60 -> 180 sec.) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index c8d6ed7d..3fc359f7 100644 --- a/requirements.yml +++ b/requirements.yml @@ -52,4 +52,4 @@ version: 4ec9187017cb7832f521fc273fabd0a873ca2736 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git - version: abeecc09f6fd165eff3cef5d6dd7696b9ca4ec27 + version: 55051deb663f7a30bf01637df7dd6b6bc2a7ce00 From 124fbeda045857233880478da814cb99444860ef Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 2 Mar 2023 22:50:13 +0200 Subject: [PATCH 280/382] Switch to using an external Etherpad role This new role also adds native Traefik support and support for other (non-`amd64`) architectures via self-building. --- CHANGELOG.md | 33 ++++- docs/configuring-playbook-etherpad.md | 69 +++++++---- group_vars/matrix_servers | 65 +++++++--- playbooks/matrix.yml | 3 +- requirements.yml | 3 + .../etherpad-proxy-connect/defaults/main.yml | 11 ++ .../tasks/inject_into_nginx_proxy.yml | 10 +- .../etherpad-proxy-connect/tasks/main.yml | 12 ++ .../tasks/validate_config.yml | 32 +++++ roles/custom/matrix-base/defaults/main.yml | 2 +- .../custom/matrix-etherpad/defaults/main.yml | 102 --------------- roles/custom/matrix-etherpad/tasks/main.yml | 29 ----- .../matrix-etherpad/tasks/setup_install.yml | 34 ----- .../matrix-etherpad/tasks/setup_uninstall.yml | 25 ---- .../matrix-etherpad/tasks/validate_config.yml | 15 --- .../templates/settings.json.j2 | 116 ------------------ .../systemd/matrix-etherpad.service.j2 | 44 ------- .../matrix-nginx-proxy/defaults/main.yml | 5 - .../matrix-nginx-proxy/templates/labels.j2 | 12 -- 19 files changed, 189 insertions(+), 433 deletions(-) create mode 100644 roles/custom/etherpad-proxy-connect/defaults/main.yml rename roles/custom/{matrix-etherpad => etherpad-proxy-connect}/tasks/inject_into_nginx_proxy.yml (81%) create mode 100644 roles/custom/etherpad-proxy-connect/tasks/main.yml create mode 100644 roles/custom/etherpad-proxy-connect/tasks/validate_config.yml delete mode 100644 roles/custom/matrix-etherpad/defaults/main.yml delete mode 100644 roles/custom/matrix-etherpad/tasks/main.yml delete mode 100644 roles/custom/matrix-etherpad/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-etherpad/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-etherpad/tasks/validate_config.yml delete mode 100644 roles/custom/matrix-etherpad/templates/settings.json.j2 delete mode 100644 roles/custom/matrix-etherpad/templates/systemd/matrix-etherpad.service.j2 diff --git a/CHANGELOG.md b/CHANGELOG.md index cd592337..790406f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,32 @@ +# 2023-03-02 + +## The matrix-etherpad role lives independently now + +**TLDR**: the `matrix-etherpad` role is now included from [another repository](https://gitlab.com/etke.cc/roles/etherpad). Some variables have been renamed. All functionality remains intact. + +You need to **update you roles** (`just roles` or `make roles`) regardless of whether you're using Etherpad or not. + +If you're making use of Etherpad via this playbook, you will need to update variable references in your `vars.yml` file: + +- Rename `matrix_etherpad_public_endpoint` to `etherpad_path_prefix` + +- Replace `matrix_etherpad_mode: dimension` with: + - for `matrix-nginx-proxy` users: + - `etherpad_nginx_proxy_dimension_integration_enabled: true` + - `etherpad_hostname: "{{ matrix_server_fqn_dimension }}"` + - for Traefik users: + - define your own `etherpad_hostname` and `etherpad_path_prefix` as you see fit + +- Rename all other variables: + - `matrix_etherpad_docker_image_` -> `matrix_etherpad_container_image_` + - `matrix_etherpad_` -> `etherpad_` + +Along with this relocation, the new role also: + +- supports [self-building](docs/self-building.md), so it should work on `arm32` and `arm64` architectures +- has native Traefik reverse-proxy support (Etherpad requests no longer go through `matrix-nginx-proxy` when using Traefik) + + # 2023-02-26 ## Traefik is the default reverse-proxy now @@ -483,11 +512,11 @@ Various services (like Dimension, etc.) still talk to Synapse via `matrix-nginx- Until now, [Etherpad](https://etherpad.org/) (which [the playbook could install for you](docs/configuring-playbook-etherpad.md)) required the [Dimension integration manager](docs/configuring-playbook-dimension.md) to also be installed, because Etherpad was hosted on the Dimension domain (at `dimension.DOMAIN/etherpad`). -From now on, Etherpad can be installed in `standalone` mode on `etherpad.DOMAIN` and used even without Dimension. This is much more versatile, so the playbook now defaults to this new mode (`matrix_etherpad_mode: standalone`). +From now on, Etherpad can be installed in `standalone` mode on `etherpad.DOMAIN` and used even without Dimension. This is much more versatile, so the playbook now defaults to this new mode (`etherpad_mode: standalone`). If you've already got both Etherpad and Dimension in use you could: -- **either** keep hosting Etherpad under the Dimension domain by adding `matrix_etherpad_mode: dimension` to your `vars.yml` file. All your existing room widgets will continue working at the same URLs and no other changes will be necessary. +- **either** keep hosting Etherpad under the Dimension domain by adding `etherpad_mode: dimension` to your `vars.yml` file. All your existing room widgets will continue working at the same URLs and no other changes will be necessary. - **or**, you could change to hosting Etherpad separately on `etherpad.DOMAIN`. You will need to [configure a DNS record](docs/configuring-dns.md) for this new domain. You will also need to reconfigure Dimension to use the new pad URLs (`https://etherpad.DOMAIN/...`) going forward (refer to our [configuring Etherpad documentation](docs/configuring-playbook-etherpad.md)). All your existing room widgets (which still use `https://dimension.DOMAIN/etherpad/...`) will break as Etherpad is not hosted there anymore. You will need to re-add them or to consider not using `standalone` mode diff --git a/docs/configuring-playbook-etherpad.md b/docs/configuring-playbook-etherpad.md index 22c782f4..76ad2f59 100644 --- a/docs/configuring-playbook-etherpad.md +++ b/docs/configuring-playbook-etherpad.md @@ -1,19 +1,41 @@ # Setting up Etherpad (optional) -[Etherpad](https://etherpad.org) is is an open source collaborative text editor that can be embedded in a Matrix chat room using the [Dimension integrations manager](https://dimension.t2bot.io) or used as standalone web app. +[Etherpad](https://etherpad.org) is an open source collaborative text editor that can be embedded in a Matrix chat room using the [Dimension integrations manager](https://dimension.t2bot.io) or used as standalone web app. When enabled together with the Jitsi audio/video conferencing system (see [our docs on Jitsi](configuring-playbook-jitsi.md)), it will be made available as an option during the conferences. -## Prerequisites +## Decide on a domain and path -Etherpad can be installed in 2 modes: +By default, Etherpad is configured to use its own dedicated domain (`etherpad.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). -- (default) `standalone` mode (`matrix_etherpad_mode: standalone`) - Etherpad will be hosted on `etherpad.` (`matrix_server_fqn_etherpad`), so the DNS record for this domian must be created. See [Configuring your DNS server](configuring-dns.md) on how to set up the `etherpad` DNS record correctly +You can override the domain and path like this: -- `dimension` mode (`matrix_etherpad_mode: dimension`) - Etherpad will be hosted on `dimension./etherpad` (`matrix_server_fqn_dimension`). This requires that you **first** configure the **Dimension integrations manager** as described in [the playbook documentation](configuring-playbook-dimension.md) +```yaml +# Switch to the domain used for Matrix services (`matrix.DOMAIN`), +# so we won't need to add additional DNS records for Etherpad. +etherpad_hostname: "{{ matrix_server_fqn_matrix }}" + +# Expose under the /etherpad subpath +etherpad_path_prefix: /etherpad +``` + +**NOTE**: When using the old `matrix-nginx-proxy` reverse-proxy instead of Traefik, you have only 2 choices: + +- serving Etherpad at its own dedicated domain: + - you need to set the domain using the `matrix_server_fqn_etherpad` variable (not `etherpad_hostname`) + - you must use `etherpad_path_prefix: /` +- serving Etherpad at the [Dimension](configuring-playbook-dimension.md) integration manager's domain (`matrix_server_fqn_dimension`) + - you need to have Dimension enabled + - you need to add `etherpad_path_prefix: /etherpad` or another prefix (different than `/`) + - you need to add `etherpad_nginx_proxy_dimension_integration_enabled: true` to enable this integration + + +## Adjusting DNS records -We recomend that you go with the default (`standalone`) mode, which makes Etherpad independent and allows it to be used with or without Dimension. +Once you've decided on the domain and path, **you may need to adjust your DNS** records to point the Etherpad domain to the Matrix server. + +If you've decided to reuse the `matrix.` domain, you won't need to do any extra DNS configuration. ## Installing @@ -21,48 +43,51 @@ We recomend that you go with the default (`standalone`) mode, which makes Etherp [Etherpad](https://etherpad.org) installation is disabled by default. You can enable it in your configuration file (`inventory/host_vars/matrix./vars.yml`): ```yaml -matrix_etherpad_enabled: true - -# Uncomment below if you'd like to install Etherpad on the Dimension domain (not recommended) -# matrix_etherpad_mode: dimension +etherpad_enabled: true # Uncomment below to enable the admin web UI -# matrix_etherpad_admin_username: admin -# matrix_etherpad_admin_password: some-password +# etherpad_admin_username: admin +# etherpad_admin_password: some-password ``` -If enabled, the admin web-UI should then be available on `https://etherpad./admin` (or `https://dimension./etherpad/admin`, if `matrix_etherpad_mode: dimension`) +Then, [run the installation process](installing.md) again (e.g. `just install-all`). + + +## Usage + +The Etherpad UI should be available at `https://etherpad.`, while the admin UI (if enabled) should then be available at `https://etherpad./admin`. + +If you've [decided on another hostname or path-prefix](#decide-on-a-domain-and-path) (e.g. `https://matrix.DOMAIN/etherpad`), adjust these URLs accordingly before usage. -## Managing / Deleting old pads +### Managing / Deleting old pads If you want to manage and remove old unused pads from Etherpad, you will first need to able Admin access as described above. -Then from the plugin manager page (`https://etherpad./admin/plugins` or `https://dimension./etherpad/admin/plugins`), install the `adminpads2` plugin. Once installed, you should have a "Manage pads" section in the Admin web-UI. +Then from the plugin manager page (`https://etherpad./admin/plugins`, install the `adminpads2` plugin. Once installed, you should have a "Manage pads" section in the Admin web-UI. -## How to use Etherpad widgets without an Integration Manager (like Dimension) +### How to use Etherpad widgets without an Integration Manager (like Dimension) This is how it works in Element, it might work quite similar with other clients: To integrate a standalone etherpad in a room, create your pad by visiting `https://etherpad.DOMAIN`. When the pad opens, copy the URL and send a command like this to the room: `/addwidget URL`. You will then find your integrated Etherpad within the right sidebar in the `Widgets` section. -## Set Dimension default to the self-hosted Etherpad (optional) +### Set Dimension default to the self-hosted Etherpad (optional) If you decided to install [Dimension integration manager](configuring-playbook-dimension.md) alongside Etherpad, the Dimension administrator users can configure the default URL template. The Dimension configuration menu can be accessed with the sprocket icon as you begin to add a widget to a room in Element. There you will find the Etherpad Widget Configuration action beneath the _Widgets_ tab. -### Removing the integrated Etherpad chat +#### Removing the integrated Etherpad chat -If you wish to disable the Etherpad chat button, you can do it by appending `?showChat=false` to the end of the pad URL, or the template. Examples: -- `https://etherpad./p/$roomId_$padName?showChat=false` (for the default - `matrix_etherpad_mode: standalone`) +If you wish to disable the Etherpad chat button, you can do it by appending `?showChat=false` to the end of the pad URL, or the template. -- `https://dimension./etherpad/p/$roomId_$padName?showChat=false` (for `matrix_etherpad_mode: dimension`) +Example: `https://etherpad./p/$roomId_$padName?showChat=false` -### Known issues +## Known issues If your Etherpad widget fails to load, this might be due to Dimension generating a Pad name so long, the Etherpad app rejects it. `$roomId_$padName` can end up being longer than 50 characters. You can avoid having this problem by altering the template so it only contains the three word random identifier `$padName`. diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6e40aba7..6acc7dde 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -304,7 +304,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-email2matrix.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'email2matrix']}] if matrix_email2matrix_enabled else []) + - ([{'name': 'matrix-etherpad.service', 'priority': 4000, 'groups': ['matrix', 'etherpad']}] if matrix_etherpad_enabled else []) + ([{'name': (etherpad_identifier + '.service'), 'priority': 4000, 'groups': ['matrix', 'etherpad']}] if etherpad_enabled else []) + ([{'name': (grafana_identifier + '.service'), 'priority': 4000, 'groups': ['matrix', 'monitoring', 'grafana']}] if grafana_enabled else []) + @@ -2208,29 +2208,55 @@ matrix_dimension_database_password: "{{ '%s' | format(matrix_homeserver_generic_ ###################################################################### # -# matrix-etherpad +# etke/etherpad # ###################################################################### -matrix_etherpad_enabled: false +etherpad_enabled: false -matrix_etherpad_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9001') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +etherpad_identifier: matrix-etherpad -matrix_etherpad_base_url: "{{ 'https://' + matrix_server_fqn_dimension + matrix_etherpad_public_endpoint if matrix_etherpad_mode == 'dimension' else 'https://' + matrix_server_fqn_etherpad + '/' }}" +etherpad_scheme: "{{ 'https' if matrix_playbook_ssl_enabled else 'http' }}" -matrix_etherpad_systemd_required_services_list: | +etherpad_base_path: "{{ matrix_base_data_path }}/etherpad" + +etherpad_framing_enabled: "{{ matrix_dimension_enabled or matrix_jitsi_enabled }}" + +etherpad_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" + +etherpad_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9001') if matrix_playbook_service_host_bind_interface_prefix else '' }}" + +etherpad_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else etherpad_identifier }}" + +etherpad_container_additional_networks: | + {{ + ( + ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + + + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != etherpad_container_network else []) + ) | unique + }} + +etherpad_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" +etherpad_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +etherpad_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +etherpad_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" + +etherpad_systemd_required_services_list: | {{ ['docker.service'] + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) }} -matrix_etherpad_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" -matrix_etherpad_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'etherpad.db', rounds=655555) | to_uuid }}" +etherpad_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" +etherpad_database_name: matrix_etherpad +etherpad_database_username: matrix_etherpad +etherpad_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'etherpad.db', rounds=655555) | to_uuid }}" ###################################################################### # -# /matrix-etherpad +# /etke/etherpad # ###################################################################### @@ -2298,9 +2324,9 @@ matrix_jitsi_web_stun_servers: | # If the self-hosted Etherpad instance is available, it will also show up in Jitsi conferences, # unless explicitly disabled by setting `matrix_jitsi_etherpad_enabled` to false. # Falls back to the scalar.vector.im etherpad in case someone sets `matrix_jitsi_etherpad_enabled` to true, -# while also setting `matrix_etherpad_enabled` to false. -matrix_jitsi_etherpad_enabled: "{{ matrix_etherpad_enabled }}" -matrix_jitsi_etherpad_base: "{{ matrix_etherpad_base_url if matrix_etherpad_enabled else 'https://scalar.vector.im/etherpad' }}" +# while also setting `etherpad_enabled` to false. +matrix_jitsi_etherpad_enabled: "{{ etherpad_enabled }}" +matrix_jitsi_etherpad_base: "{{ etherpad_base_url if etherpad_enabled else 'https://scalar.vector.im/etherpad' }}" # Allow verification using JWT and matrix-UVS matrix_jitsi_prosody_auth_matrix_uvs_auth_token: "{{ matrix_user_verification_service_uvs_auth_token }}" @@ -2468,7 +2494,7 @@ matrix_nginx_proxy_proxy_cinny_enabled: "{{ matrix_client_cinny_enabled and matr matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" -matrix_nginx_proxy_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" +matrix_nginx_proxy_proxy_etherpad_enabled: "{{ etherpad_enabled and not etherpad_nginx_proxy_dimension_integration_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" @@ -2484,7 +2510,6 @@ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_ent matrix_nginx_proxy_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true -matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: "{{ matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone' }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" @@ -2576,7 +2601,7 @@ matrix_nginx_proxy_systemd_wanted_services_list: | + (['matrix-bot-go-neb.service'] if matrix_bot_go_neb_enabled else []) + - (['matrix-etherpad.service'] if matrix_etherpad_enabled else []) + ([etherpad_identifier + '.service'] if etherpad_enabled else []) + (['matrix-hookshot.service'] if matrix_hookshot_enabled else []) }} @@ -2597,7 +2622,7 @@ matrix_ssl_domains_to_obtain_certificates_for: | + ([matrix_server_fqn_dimension] if matrix_dimension_enabled else []) + - ([matrix_server_fqn_etherpad] if (matrix_etherpad_enabled and matrix_etherpad_mode == 'standalone') else []) + ([matrix_server_fqn_etherpad] if (etherpad_enabled and not etherpad_nginx_proxy_dimension_integration_enabled) else []) + ([matrix_server_fqn_bot_go_neb] if matrix_bot_go_neb_enabled else []) + @@ -2888,10 +2913,10 @@ devture_postgres_managed_databases_auto: | }] if (matrix_dimension_enabled and matrix_dimension_database_engine == 'postgres' and matrix_dimension_database_hostname == devture_postgres_connection_hostname) else []) + ([{ - 'name': matrix_etherpad_database_name, - 'username': matrix_etherpad_database_username, - 'password': matrix_etherpad_database_password, - }] if (matrix_etherpad_enabled and matrix_etherpad_database_engine == 'postgres' and matrix_etherpad_database_hostname == devture_postgres_connection_hostname) else []) + 'name': etherpad_database_name, + 'username': etherpad_database_username, + 'password': etherpad_database_password, + }] if (etherpad_enabled and etherpad_database_engine == 'postgres' and etherpad_database_hostname == devture_postgres_connection_hostname) else []) + ([{ 'name': prometheus_postgres_exporter_database_name, diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 6a8987bf..6a66dc58 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -103,7 +103,8 @@ - custom/matrix-ldap-registration-proxy - custom/matrix-ma1sd - custom/matrix-dimension - - custom/matrix-etherpad + - galaxy/etherpad + - custom/etherpad-proxy-connect - custom/matrix-email2matrix - custom/matrix-sygnal - galaxy/ntfy diff --git a/requirements.yml b/requirements.yml index 3fc359f7..89d26802 100644 --- a/requirements.yml +++ b/requirements.yml @@ -48,6 +48,9 @@ - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.9-0 +- src: git+https://gitlab.com/etke.cc/roles/etherpad.git + version: v1.8.18-0 + - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: 4ec9187017cb7832f521fc273fabd0a873ca2736 diff --git a/roles/custom/etherpad-proxy-connect/defaults/main.yml b/roles/custom/etherpad-proxy-connect/defaults/main.yml new file mode 100644 index 00000000..83df826f --- /dev/null +++ b/roles/custom/etherpad-proxy-connect/defaults/main.yml @@ -0,0 +1,11 @@ +--- + +# etherpad-proxy-connect is a compatibility role connecting the new Etherpad role with matrix-nginx-proxy. +# It adds back support for serving Etherpad under the Dimension domain (`matrix_server_fqn_dimension`). + +# Controls whether Etherpad will be hosted under the Dimension domain when matrix-nginx-proxy is used (depending on matrix_playbook_reverse_proxy_type). +# If you're not using matrix-nginx-proxy, then this value has no effect. +etherpad_nginx_proxy_dimension_integration_enabled: false + +# Controls the path at which Etherpad will be exposed on the Dimension domain. +etherpad_nginx_proxy_dimension_integration_path_prefix: "{{ etherpad_path_prefix }}" diff --git a/roles/custom/matrix-etherpad/tasks/inject_into_nginx_proxy.yml b/roles/custom/etherpad-proxy-connect/tasks/inject_into_nginx_proxy.yml similarity index 81% rename from roles/custom/matrix-etherpad/tasks/inject_into_nginx_proxy.yml rename to roles/custom/etherpad-proxy-connect/tasks/inject_into_nginx_proxy.yml index e32cdf96..6252dba1 100644 --- a/roles/custom/matrix-etherpad/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/etherpad-proxy-connect/tasks/inject_into_nginx_proxy.yml @@ -11,14 +11,14 @@ - name: Generate Etherpad proxying configuration for matrix-nginx-proxy ansible.builtin.set_fact: - matrix_etherpad_matrix_nginx_proxy_configuration: | - rewrite ^{{ matrix_etherpad_public_endpoint }}$ {{ matrix_nginx_proxy_x_forwarded_proto_value }}://$server_name{{ matrix_etherpad_public_endpoint }}/ permanent; + etherpad_matrix_nginx_proxy_configuration: | + rewrite ^{{ etherpad_nginx_proxy_dimension_integration_path_prefix }}$ {{ matrix_nginx_proxy_x_forwarded_proto_value }}://$server_name{{ etherpad_nginx_proxy_dimension_integration_path_prefix }}/ permanent; - location {{ matrix_etherpad_public_endpoint }}/ { + location {{ etherpad_nginx_proxy_dimension_integration_path_prefix }}/ { {% if matrix_nginx_proxy_enabled | default(False) %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; - proxy_pass http://matrix-etherpad:9001/; + proxy_pass http://{{ etherpad_identifier }}:9001/; {# These are proxy directives needed specifically by Etherpad #} proxy_buffering off; proxy_http_version 1.1; # recommended with keepalive connections @@ -42,5 +42,5 @@ {{ matrix_nginx_proxy_proxy_dimension_additional_server_configuration_blocks | default([]) + - [matrix_etherpad_matrix_nginx_proxy_configuration] + [etherpad_matrix_nginx_proxy_configuration] }} diff --git a/roles/custom/etherpad-proxy-connect/tasks/main.yml b/roles/custom/etherpad-proxy-connect/tasks/main.yml new file mode 100644 index 00000000..67bf7d0e --- /dev/null +++ b/roles/custom/etherpad-proxy-connect/tasks/main.yml @@ -0,0 +1,12 @@ +--- + +- when: etherpad_enabled | bool and etherpad_nginx_proxy_dimension_integration_enabled | bool + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - install-all + - setup-all + - install-nginx-proxy + - setup-nginx-proxy diff --git a/roles/custom/etherpad-proxy-connect/tasks/validate_config.yml b/roles/custom/etherpad-proxy-connect/tasks/validate_config.yml new file mode 100644 index 00000000..2127ce1a --- /dev/null +++ b/roles/custom/etherpad-proxy-connect/tasks/validate_config.yml @@ -0,0 +1,32 @@ +--- + +- when: matrix_playbook_reverse_proxy_type not in ['playbook-managed-nginx', 'other-nginx-non-container'] + name: Fail if reverse-proxy is not nginx + ansible.builtin.fail: + msg: > + Etherpad's integration into matrix-nginx-proxy's Dimension server only makes sense if you're using matrix-nginx-proxy. + `matrix_playbook_reverse_proxy_type` ({{ matrix_playbook_reverse_proxy_type }}) indicates that you're using another reverse-proxy. + If you're using Traefik, you should configure `etherpad_hostname` and `etherpad_path_prefix` instead. + +- when: not matrix_dimension_enabled + name: Fail if Dimension not enabled + ansible.builtin.fail: + msg: > + Etherpad's integration into matrix-nginx-proxy's Dimension server only makes sense if you're using Dimension. + Looks like Dimension is not enabled in your configuration (judging by `matrix_dimension_enabled`). + Consider configuring `etherpad_hostname` and `etherpad_path_prefix` instead. + +- when: etherpad_hostname != matrix_server_fqn_dimension + name: Fail if Etherpad hostname does not match Dimension hostname + ansible.builtin.fail: + msg: > + Etherpad's integration into matrix-nginx-proxy's Dimension server requires that you set `etherpad_hostname` to `matrix_server_fqn_dimension`. + Consider adding this to your configuration: `{% raw %}etherpad_hostname: "{{ matrix_server_fqn_dimension }}"{% endraw %}` + +- when: etherpad_nginx_proxy_dimension_integration_path_prefix == '/' + name: Fail if / path prefix used for Etherpad + ansible.builtin.fail: + msg: > + Etherpad's integration into matrix-nginx-proxy's Dimension server only makes sense if you're using a non-`/` path for Etherpad. + You've chosen a path prefix of `/` in `etherpad_nginx_proxy_dimension_integration_path_prefix`. + The `/` path must go to Dimension itself, so you need to pick a different prefix (e.g. `/etherpad`). diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 403af983..98a2167e 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -72,7 +72,7 @@ matrix_server_fqn_buscarron: "buscarron.{{ matrix_domain }}" # This is where you access the Dimension. matrix_server_fqn_dimension: "dimension.{{ matrix_domain }}" -# This is where you access the etherpad (if enabled via matrix_etherpad_enabled; disabled by default). +# This is where you access the etherpad (if enabled via etherpad_enabled; disabled by default). matrix_server_fqn_etherpad: "etherpad.{{ matrix_domain }}" # For use with Go-NEB! (github callback url for example) diff --git a/roles/custom/matrix-etherpad/defaults/main.yml b/roles/custom/matrix-etherpad/defaults/main.yml deleted file mode 100644 index 874e9a30..00000000 --- a/roles/custom/matrix-etherpad/defaults/main.yml +++ /dev/null @@ -1,102 +0,0 @@ ---- -# Project source code URL: https://github.com/ether/etherpad-lite - -matrix_etherpad_enabled: false - -# standalone = etherpad installed on subdomain (etherpad.DOMAIN) and can be used as-is -# dimension = etherpad installed in subdir of dimension (dimension.DOMAIN/etherpad) and can be used with dimension -matrix_etherpad_mode: standalone - -matrix_etherpad_base_path: "{{ matrix_base_data_path }}/etherpad" - -matrix_etherpad_version: 1.8.18 -matrix_etherpad_docker_image: "{{ matrix_container_global_registry_prefix }}etherpad/etherpad:{{ matrix_etherpad_version }}" -matrix_etherpad_docker_image_force_pull: "{{ matrix_etherpad_docker_image.endswith(':latest') }}" - -# List of systemd services that matrix-etherpad.service depends on. -matrix_etherpad_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-etherpad.service wants -matrix_etherpad_systemd_wanted_services_list: [] - -# Container user has to be able to write to the source file directories until this bug is fixed: -# https://github.com/ether/etherpad-lite/issues/2683 -matrix_etherpad_user_uid: '5001' -matrix_etherpad_user_gid: '5001' - -# Controls whether the matrix-etherpad container exposes its HTTP port (tcp/9001 in the container). -# -# Takes an ":" or "" value (e.g. "127.0.0.1:9001"), or empty string to not expose. -matrix_etherpad_container_http_host_bind_port: '' - -# A list of extra arguments to pass to the container -matrix_etherpad_container_extra_arguments: [] - -# Used only when `matrix_etherpad_mode: dimension` -matrix_etherpad_public_endpoint: '/etherpad' - -# By default, the Etherpad app can be accessed on etherpad subdomain -matrix_etherpad_base_url: "https://{{ matrix_server_fqn_etherpad }}/" - -# Database-related configuration fields. -# -# Etherpad requires a dedicated database -matrix_etherpad_database_engine: 'postgres' - -matrix_etherpad_database_username: 'matrix_etherpad' -matrix_etherpad_database_password: 'some-password' -matrix_etherpad_database_hostname: '' -matrix_etherpad_database_port: 5432 -matrix_etherpad_database_name: 'matrix_etherpad' - -# If a admin username and password is set, the /admin web page will be -# available. -matrix_etherpad_admin_username: '' -matrix_etherpad_admin_password: '' - -matrix_etherpad_database_connection_string: 'postgres://{{ matrix_etherpad_database_username }}:{{ matrix_etherpad_database_password }}@{{ matrix_etherpad_database_hostname }}:{{ matrix_etherpad_database_port }}/{{ matrix_etherpad_database_name }}' - -# Variables configuring the etherpad -matrix_etherpad_title: 'Etherpad' -matrix_etherpad_abiword: null -matrix_etherpad_soffice: null -matrix_etherpad_default_pad_text: | - Welcome to Etherpad! - - This pad text is synchronized as you type, so that everyone viewing this page sees the same text. This allows you to collaborate seamlessly on documents! - - Get involved with Etherpad at https://etherpad.org - -# Default Etherpad configuration template which covers the generic use case. -# You can customize it by controlling the various variables inside it. -# -# For a more advanced customization, you can extend the default (see `matrix_etherpad_configuration_extension_json`) -# or completely replace this variable with your own template. -matrix_etherpad_configuration_default: "{{ lookup('template', 'templates/settings.json.j2') }}" - -# Your custom JSON configuration for Etherpad goes here. -# This configuration extends the default starting configuration (`matrix_etherpad_configuration_json`). -# -# You can override individual variables from the default configuration, or introduce new ones. -# -# If you need something more special, you can take full control by -# completely redefining `matrix_etherpad_configuration_json`. -# -# Example configuration extension follows: -# -# matrix_etherpad_configuration_extension_json: | -# { -# "loadTest": true, -# "commitRateLimiting": { -# "duration": 1, -# "points": 10 -# } -# } -# -matrix_etherpad_configuration_extension_json: '{}' - -matrix_etherpad_configuration_extension: "{{ matrix_etherpad_configuration_extension_json | from_json if matrix_etherpad_configuration_extension_json | from_json is mapping else {} }}" - -# Holds the final Etherpad configuration (a combination of the default and its extension). -# You most likely don't need to touch this variable. Instead, see `matrix_etherpad_configuration_json`. -matrix_etherpad_configuration: "{{ matrix_etherpad_configuration_default | combine(matrix_etherpad_configuration_extension, recursive=True) }}" diff --git a/roles/custom/matrix-etherpad/tasks/main.yml b/roles/custom/matrix-etherpad/tasks/main.yml deleted file mode 100644 index eaafd98a..00000000 --- a/roles/custom/matrix-etherpad/tasks/main.yml +++ /dev/null @@ -1,29 +0,0 @@ ---- - -- block: - - when: matrix_etherpad_enabled | bool and matrix_etherpad_mode == 'dimension' - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: - - setup-all - - setup-nginx-proxy - - install-all - - install-nginx-proxy - -- block: - - when: matrix_etherpad_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - - - when: matrix_etherpad_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-etherpad - - install-all - - install-etherpad - -- block: - - when: not matrix_etherpad_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-etherpad diff --git a/roles/custom/matrix-etherpad/tasks/setup_install.yml b/roles/custom/matrix-etherpad/tasks/setup_install.yml deleted file mode 100644 index d4e2952e..00000000 --- a/roles/custom/matrix-etherpad/tasks/setup_install.yml +++ /dev/null @@ -1,34 +0,0 @@ ---- - -- name: Ensure Etherpad base path exists - ansible.builtin.file: - path: "{{ matrix_etherpad_base_path }}" - state: directory - mode: 0770 - owner: "{{ matrix_etherpad_user_uid }}" - group: "{{ matrix_etherpad_user_gid }}" - -- name: Ensure Etherpad config installed - ansible.builtin.copy: - content: "{{ matrix_etherpad_configuration | to_nice_json }}" - dest: "{{ matrix_etherpad_base_path }}/settings.json" - mode: 0640 - owner: "{{ matrix_etherpad_user_uid }}" - group: "{{ matrix_etherpad_user_gid }}" - -- name: Ensure Etherpad image is pulled - community.docker.docker_image: - name: "{{ matrix_etherpad_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_etherpad_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_etherpad_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure matrix-etherpad.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-etherpad.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-etherpad.service" - mode: 0644 diff --git a/roles/custom/matrix-etherpad/tasks/setup_uninstall.yml b/roles/custom/matrix-etherpad/tasks/setup_uninstall.yml deleted file mode 100644 index e0382984..00000000 --- a/roles/custom/matrix-etherpad/tasks/setup_uninstall.yml +++ /dev/null @@ -1,25 +0,0 @@ ---- - -- name: Check existence of matrix-etherpad service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-etherpad.service" - register: matrix_etherpad_service_stat - -- when: matrix_etherpad_service_stat.stat.exists | bool - block: - - name: Ensure matrix-etherpad is stopped - ansible.builtin.service: - name: matrix-etherpad - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-etherpad.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-etherpad.service" - state: absent - - - name: Ensure Etherpad base directory doesn't exist - ansible.builtin.file: - path: "{{ matrix_etherpad_base_path }}" - state: absent diff --git a/roles/custom/matrix-etherpad/tasks/validate_config.yml b/roles/custom/matrix-etherpad/tasks/validate_config.yml deleted file mode 100644 index 2bc773a6..00000000 --- a/roles/custom/matrix-etherpad/tasks/validate_config.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- - -- name: Fail if required Etherpad settings not defined - ansible.builtin.fail: - msg: > - You need to define a required configuration setting (`{{ item.name }}`). - when: "item.when | bool and vars[item.name] == ''" - with_items: - - {'name': 'matrix_etherpad_database_hostname', when: true} - -- name: Fail if wrong mode selected - ansible.builtin.fail: - msg: >- - You're using Etherpad in 'dimension' mode (`matrix_etherpad_serving_mode: dimension`), which tries to host Etherpad at the Dimension subdomain - `{{ matrix_server_fqn_dimension }}`. However, this isn't possible because Dimension is not enabled. To resolve the problem, either enable Dimension (`matrix_dimension_enabled: true`) or switch Etherpad to standalone mode (`matrix_etherpad_mode: standalone`) and have it served on its own domain (`{{ matrix_server_fqn_etherpad }}`). - when: matrix_etherpad_enabled | bool and matrix_etherpad_mode == 'dimension' and not matrix_dimension_enabled | default(False) | bool diff --git a/roles/custom/matrix-etherpad/templates/settings.json.j2 b/roles/custom/matrix-etherpad/templates/settings.json.j2 deleted file mode 100644 index cc45d190..00000000 --- a/roles/custom/matrix-etherpad/templates/settings.json.j2 +++ /dev/null @@ -1,116 +0,0 @@ -{ - "title": {{ matrix_etherpad_title|to_json }}, - "favicon": "favicon.ico", - "skinName": "colibris", - "skinVariants": "super-light-toolbar super-light-editor light-background", - "ip": "::", - "port": 9001, - "showSettingsInAdminPage": true, - "dbType": {{ matrix_etherpad_database_engine|to_json }}, - "dbSettings": { - "database": {{ matrix_etherpad_database_name|to_json }}, - "host": {{ matrix_etherpad_database_hostname|to_json }}, - "password": {{ matrix_etherpad_database_password|to_json }}, - "port": {{ matrix_etherpad_database_port|to_json }}, - "user": {{ matrix_etherpad_database_username|to_json }} - }, - "defaultPadText" : {{ matrix_etherpad_default_pad_text|to_json }}, - "suppressErrorsInPadText": false, - "requireSession": false, - "editOnly": false, - "minify": true, - "maxAge": 21600, - "abiword": {{ matrix_etherpad_abiword|to_json }}, - "soffice": {{ matrix_etherpad_soffice|to_json }}, - "tidyHtml": null, - "allowUnknownFileEnds": true, - "requireAuthentication": false, - "requireAuthorization": false, - "trustProxy": true, - "cookie": { - "sameSite": "Lax" - }, - "disableIPlogging": true, - "automaticReconnectionTimeout": 0, - "scrollWhenFocusLineIsOutOfViewport": { - "percentage": { - "editionAboveViewport": 0, - "editionBelowViewport": 0 - }, - "duration": 0, - "scrollWhenCaretIsInTheLastLineOfViewport": false, - "percentageToScrollWhenUserPressesArrowUp": 0 - }, - "socketTransportProtocols" : ["xhr-polling", "jsonp-polling", "htmlfile"], - "socketIo": { - "maxHttpBufferSize": 10000 - }, - "loadTest": false, - "dumpOnUncleanExit": false, - "indentationOnNewLine": false, - "importExportRateLimiting": { - "windowMs": 90000, - "max": 10 - }, - "importMaxFileSize": 52428800, - "commitRateLimiting": { - "duration": 1, - "points": 10 - }, - "exposeVersion": false, - "padOptions": { - "noColors": false, - "showControls": true, - "showChat": false, - "showLineNumbers": true, - "useMonospaceFont": false, - "userName": false, - "userColor": false, - "rtl": false, - "alwaysShowChat": false, - "chatAndUsers": false, - "lang": "en-gb" - }, -{% if matrix_etherpad_admin_username != "" and matrix_etherpad_admin_password != "" %} - "users": { - {{ matrix_etherpad_admin_username|to_json }}: { - "password": {{ matrix_etherpad_admin_password|to_json }}, - "is_admin": true - } - }, -{% endif %} - "padShortcutEnabled" : { - "altF9": true, - "altC": true, - "cmdShift2": true, - "delete": true, - "return": true, - "esc": true, - "cmdS": true, - "tab": true, - "cmdZ": true, - "cmdY": true, - "cmdI": true, - "cmdB": true, - "cmdU": true, - "cmd5": true, - "cmdShiftL": true, - "cmdShiftN": true, - "cmdShift1": true, - "cmdShiftC": true, - "cmdH": true, - "ctrlHome": true, - "pageUp": true, - "pageDown": true - }, - "loglevel": "WARN", - "logconfig" : - { "appenders": [ - { "type": "console", - "layout": {"type": "messagePassThrough"} - } - ] - }, - "customLocaleStrings": {}, - "enableAdminUITests": false -} diff --git a/roles/custom/matrix-etherpad/templates/systemd/matrix-etherpad.service.j2 b/roles/custom/matrix-etherpad/templates/systemd/matrix-etherpad.service.j2 deleted file mode 100644 index e0a10481..00000000 --- a/roles/custom/matrix-etherpad/templates/systemd/matrix-etherpad.service.j2 +++ /dev/null @@ -1,44 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=Matrix Etherpad -{% for service in matrix_etherpad_systemd_required_services_list %} -Requires={{ service }} -After={{ service }} -{% endfor %} -{% for service in matrix_etherpad_systemd_wanted_services_list %} -Wants={{ service }} -{% endfor %} -DefaultDependencies=no - -[Service] -Type=simple -Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} kill matrix-etherpad -ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} rm matrix-etherpad - -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-etherpad \ - --log-driver=none \ - --user={{ matrix_etherpad_user_uid }}:{{ matrix_etherpad_user_gid }} \ - --cap-drop=ALL \ - --network={{ matrix_docker_network }} \ - {% if matrix_etherpad_container_http_host_bind_port %} - -p {{ matrix_etherpad_container_http_host_bind_port }}:9001 \ - {% endif %} - --mount type=bind,src={{ matrix_etherpad_base_path }},dst=/data \ - {% for arg in matrix_etherpad_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - {{ matrix_etherpad_docker_image }} \ - node --experimental-worker src/node/server.js \ - --settings /data/settings.json --credentials /data/credentials.json \ - --sessionkey /data/sessionkey.json --apikey /data/apijey.json - - -ExecStop=-{{ devture_systemd_docker_base_host_command_docker }} kill matrix-etherpad -ExecStop=-{{ devture_systemd_docker_base_host_command_docker }} rm matrix-etherpad -Restart=always -RestartSec=30 -SyslogIdentifier=matrix-etherpad - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 27978cae..daabcdb9 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -63,11 +63,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_rule: "Host( matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" -matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_hostname: "{{ matrix_server_fqn_etherpad }}" -matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_hostname: "{{ matrix_server_fqn_bot_go_neb }}" matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 0aff6a05..4cf78b2e 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -39,18 +39,6 @@ traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints={{ matrix_ {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_enabled %} -# Etherpad -traefik.http.routers.matrix-nginx-proxy-etherpad.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_rule }} -traefik.http.routers.matrix-nginx-proxy-etherpad.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-etherpad.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_etherpad_tls %} -traefik.http.routers.matrix-nginx-proxy-etherpad.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-etherpad.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - {% if matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled %} # Go NEB bot traefik.http.routers.matrix-nginx-proxy-bot_go_neb.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_rule }} From 795c3353958a9877616faf0c49c0bd09ee96ff9e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 2 Mar 2023 22:58:45 +0200 Subject: [PATCH 281/382] Upgrade Etherpad role (v1.8.18-0 -> v1.8.18-1) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 89d26802..d3fdefff 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.9-0 - src: git+https://gitlab.com/etke.cc/roles/etherpad.git - version: v1.8.18-0 + version: v1.8.18-1 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: 4ec9187017cb7832f521fc273fabd0a873ca2736 From 849248b165b2e1b8a35babe9f49f5ebf05935a71 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 2 Mar 2023 23:00:18 +0200 Subject: [PATCH 282/382] Upgrade Etherpad role (v1.8.18-1 -> v1.8.18-2) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index d3fdefff..dc1c9d56 100644 --- a/requirements.yml +++ b/requirements.yml @@ -49,7 +49,7 @@ version: v7.0.9-0 - src: git+https://gitlab.com/etke.cc/roles/etherpad.git - version: v1.8.18-1 + version: v1.8.18-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: 4ec9187017cb7832f521fc273fabd0a873ca2736 From 47cfec726f1e640fe28dbb28f12ef7c34e01af23 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 3 Mar 2023 06:18:52 +0000 Subject: [PATCH 283/382] update grafana 9.4.1 -> 9.4.2 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index dc1c9d56..025c285c 100644 --- a/requirements.yml +++ b/requirements.yml @@ -40,7 +40,7 @@ version: v1.2.3-1.7.7-1 - src: git+https://gitlab.com/etke.cc/roles/grafana.git - version: v9.4.1-0 + version: v9.4.2-0 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git version: v2.1.1-0 From bb19de4a5f63e47fadd46aba65802338b69b6fd9 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 3 Mar 2023 06:21:26 +0000 Subject: [PATCH 284/382] update borgmatic 1.7.7 -> 1.7.8 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index dc1c9d56..4ffbce13 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git - version: v1.2.3-1.7.7-1 + version: v1.2.3-1.7.8-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git version: v9.4.1-0 From 49b7d805ee3d03893a3208f0bef9b9d4fc8452a2 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 3 Mar 2023 08:58:23 +0200 Subject: [PATCH 285/382] Upgrade com.devture.ansible.role.traefik --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 025c285c..edb58785 100644 --- a/requirements.yml +++ b/requirements.yml @@ -52,7 +52,7 @@ version: v1.8.18-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: 4ec9187017cb7832f521fc273fabd0a873ca2736 + version: v2.9.8-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: 55051deb663f7a30bf01637df7dd6b6bc2a7ce00 From 0dcfc74fc83b947ad2f841940ed7b31e86c2ba9e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 3 Mar 2023 08:59:57 +0200 Subject: [PATCH 286/382] Upgrade com.devture.ansible.role.traefik_certs_dumper --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index edb58785..dbe3ded9 100644 --- a/requirements.yml +++ b/requirements.yml @@ -55,4 +55,4 @@ version: v2.9.8-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git - version: 55051deb663f7a30bf01637df7dd6b6bc2a7ce00 + version: v2.8.1-0 From 70b67b12bcf60d89efc19ea7cce4f7c63d3df955 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 3 Mar 2023 09:04:13 +0200 Subject: [PATCH 287/382] Upgrade com.devture.ansible.role.postgres_backup --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index dbe3ded9..b2fd25ec 100644 --- a/requirements.yml +++ b/requirements.yml @@ -22,7 +22,7 @@ version: a1bb78d194434b38005f3a9e623bfa4b2c06c7bc - src: git+https://github.com/devture/com.devture.ansible.role.postgres_backup.git - version: 77b1f9ae1aafa31c9078178c1036bf744c99d08b + version: 8e9ec48a09284c84704d7a2dce17da35f181574d - src: git+https://github.com/devture/com.devture.ansible.role.systemd_service_manager.git version: 6ccb88ac5fc27e1e70afcd48278ade4b564a9096 From f8966cd8da533516aa1466fb2838045c4c1200cc Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 3 Mar 2023 09:47:05 +0200 Subject: [PATCH 288/382] Default etherpad_hostname to matrix_server_fqn_etherpad for backward compatibility --- group_vars/matrix_servers | 2 ++ 1 file changed, 2 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6acc7dde..7f1a729f 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2222,6 +2222,8 @@ etherpad_base_path: "{{ matrix_base_data_path }}/etherpad" etherpad_framing_enabled: "{{ matrix_dimension_enabled or matrix_jitsi_enabled }}" +etherpad_hostname: "{{ matrix_server_fqn_etherpad }}" + etherpad_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" etherpad_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9001') if matrix_playbook_service_host_bind_interface_prefix else '' }}" From e588f5eaec2329c1217b54a709cfdf5385067638 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 3 Mar 2023 07:47:47 +0000 Subject: [PATCH 289/382] update grafana 9.4.2 -> 9.4.3 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 5154cc73..3dd08913 100644 --- a/requirements.yml +++ b/requirements.yml @@ -40,7 +40,7 @@ version: v1.2.3-1.7.8-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git - version: v9.4.2-0 + version: v9.4.3-0 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git version: v2.1.1-0 From 6085e3a8163cb497ed27e1002d5cd02438a736fe Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 3 Mar 2023 10:37:51 +0200 Subject: [PATCH 290/382] Add validation tasks for Etherpad migration (matrix_etherpad -> etherpad) --- .../matrix_playbook_migration/defaults/main.yml | 3 +++ .../tasks/validate_config.yml | 15 +++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml index 514125ec..dbc3b3b8 100644 --- a/roles/custom/matrix_playbook_migration/defaults/main.yml +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -17,3 +17,6 @@ matrix_playbook_migration_matrix_ntfy_migration_validation_enabled: true # Controls if (`matrix_redis` -> `redis`) validation will run. matrix_playbook_migration_matrix_redis_migration_validation_enabled: true + +# Controls if (`matrix_etherpad` -> `etherpad`) validation will run. +matrix_playbook_migration_matrix_etherpad_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 2241d502..296f2daf 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -141,3 +141,18 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_redis_` -> `redis_`). We found usage of the following variables: {{ matrix_playbook_migration_redis_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_redis_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_etherpad_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_etherpad_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_etherpad_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_etherpad variables + ansible.builtin.fail: + msg: >- + The matrix-etherpad role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/etherpad. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_etherpad_` -> `etherpad_`). + We found usage of the following variables: {{ matrix_playbook_migration_etherpad_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_etherpad_migration_vars | length > 0" From 10b53503704d192d29008745f7d9c28e38f58e25 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 3 Mar 2023 10:38:38 +0200 Subject: [PATCH 291/382] Add Traefik support to Go-NEB bot Completely untested. --- docs/configuring-playbook-bot-go-neb.md | 29 ++++- group_vars/matrix_servers | 31 ++++- .../matrix-bot-go-neb/defaults/main.yml | 118 +++++++++++++++++- .../tasks/{setup_install.yml => install.yml} | 49 ++++---- roles/custom/matrix-bot-go-neb/tasks/main.yml | 4 +- .../{setup_uninstall.yml => uninstall.yml} | 0 .../custom/matrix-bot-go-neb/templates/env.j2 | 8 ++ .../matrix-bot-go-neb/templates/labels.j2 | 45 +++++++ .../systemd/matrix-bot-go-neb.service.j2 | 22 ++-- .../matrix-nginx-proxy/defaults/main.yml | 5 - .../matrix-nginx-proxy/templates/labels.j2 | 12 -- 11 files changed, 265 insertions(+), 58 deletions(-) rename roles/custom/matrix-bot-go-neb/tasks/{setup_install.yml => install.yml} (63%) rename roles/custom/matrix-bot-go-neb/tasks/{setup_uninstall.yml => uninstall.yml} (100%) create mode 100644 roles/custom/matrix-bot-go-neb/templates/env.j2 create mode 100644 roles/custom/matrix-bot-go-neb/templates/labels.j2 diff --git a/docs/configuring-playbook-bot-go-neb.md b/docs/configuring-playbook-bot-go-neb.md index 6ec2056c..77351de2 100644 --- a/docs/configuring-playbook-bot-go-neb.md +++ b/docs/configuring-playbook-bot-go-neb.md @@ -24,6 +24,31 @@ ansible-playbook -i inventory/hosts setup.yml --extra-vars='username=bot.go-neb Once the user is created you can [obtain an access token](obtaining-access-tokens.md). +## Decide on a domain and path + +By default, Go-NEB is configured to use its own dedicated domain (`goneb.DOMAIN`) and requires you to [adjust your DNS records](#adjusting-dns-records). + +You can override the domain and path like this: + +```yaml +# Switch to the domain used for Matrix services (`matrix.DOMAIN`), +# so we won't need to add additional DNS records for Go-NEB. +matrix_bot_go_neb_hostname: "{{ matrix_server_fqn_matrix }}" + +# Expose under the /go-neb subpath +matrix_bot_go_neb_path_prefix: /go-neb +``` + +**NOTE**: When using `matrix-nginx-proxy` instead of Traefik, you won't be able to override the path prefix. You can only override the domain, but that needs to happen using another variable: `matrix_server_fqn_go_neb` (e.g. `matrix_server_fqn_go_neb: "mybot.{{ matrix_domain }}"`). + + +## Adjusting DNS records + +Once you've decided on the domain and path, **you may need to adjust your DNS** records to point the Go-NEB domain to the Matrix server. + +If you've decided to reuse the `matrix.` domain, you won't need to do any extra DNS configuration. + + ## Adjusting the playbook configuration Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file (adapt to your needs): @@ -193,9 +218,7 @@ matrix_bot_go_neb_services: ## Installing -Don't forget to add `goneb.` to DNS as described in [Configuring DNS](configuring-dns.md) before running the playbook. - -After configuring the playbook, run the [installation](installing.md) command again: +After potentially [adjusting DNS records](#adjusting-dns-records) and configuring the playbook, run the [installation](installing.md) command again: ``` ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 7f1a729f..0a3ff00b 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1829,17 +1829,41 @@ matrix_bot_chatgpt_container_image_self_build: "{{ matrix_architecture not in [' # We don't enable bots by default. matrix_bot_go_neb_enabled: false +matrix_bot_go_neb_scheme: "{{ 'https' if matrix_playbook_ssl_enabled else 'http' }}" + +matrix_bot_go_neb_hostname: "{{ matrix_server_fqn_bot_go_neb }}" + +matrix_bot_go_neb_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-bot-go-neb' }}" + +matrix_bot_go_neb_container_additional_networks: | + {{ + ( + ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + + + ([matrix_homeserver_container_network] if matrix_homeserver_container_network != matrix_bot_go_neb_container_network else []) + + + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != matrix_bot_go_neb_container_network else []) + ) | unique + }} + +matrix_bot_go_neb_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '4050') if matrix_playbook_service_host_bind_interface_prefix else '' }}" + +matrix_bot_go_neb_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +matrix_bot_go_neb_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_bot_go_neb_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_bot_go_neb_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" + matrix_bot_go_neb_systemd_required_services_list: | {{ ['docker.service'] + ['matrix-' + matrix_homeserver_implementation + '.service'] + + ([devture_postgres_identifier + '.service'] if devture_postgres_enabled else []) + + (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) }} -matrix_bot_go_neb_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '4050') if matrix_playbook_service_host_bind_interface_prefix else '' }}" - ###################################################################### # # /matrix-bot-go-neb @@ -2497,7 +2521,7 @@ matrix_nginx_proxy_proxy_buscarron_enabled: "{{ matrix_bot_buscarron_enabled and matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_rageshake_enabled: "{{ matrix_rageshake_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_etherpad_enabled: "{{ etherpad_enabled and not etherpad_nginx_proxy_dimension_integration_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" -matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" +matrix_nginx_proxy_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled and matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] }}" matrix_nginx_proxy_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" matrix_nginx_proxy_proxy_jitsi_manage_wellknown: "{{ matrix_jitsi_require_well_known }}" @@ -2512,7 +2536,6 @@ matrix_nginx_proxy_container_labels_traefik_entrypoints: "{{ devture_traefik_ent matrix_nginx_proxy_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_enabled: true -matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: "{{ matrix_bot_go_neb_enabled }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: "{{ matrix_corporal_enabled and matrix_corporal_http_api_enabled }}" diff --git a/roles/custom/matrix-bot-go-neb/defaults/main.yml b/roles/custom/matrix-bot-go-neb/defaults/main.yml index 61c5d54d..39b97b08 100644 --- a/roles/custom/matrix-bot-go-neb/defaults/main.yml +++ b/roles/custom/matrix-bot-go-neb/defaults/main.yml @@ -1,11 +1,22 @@ --- + # Go-NEB is a Matrix bot written in Go. It is the successor to Matrix-NEB, the original Matrix bot written in Python. # Project source code URL: https://github.com/matrix-org/go-neb matrix_bot_go_neb_enabled: true + matrix_bot_go_neb_version: latest -matrix_bot_go_neb_docker_image: "matrixdotorg/go-neb:{{ matrix_bot_go_neb_version }}" -matrix_bot_go_neb_docker_image_force_pull: "{{ matrix_bot_go_neb_docker_image.endswith(':latest') }}" + +matrix_bot_go_neb_scheme: https + +# The hostname at which Go-NEB is served. +matrix_bot_go_neb_hostname: '' + +# The path at which Go-NEB is exposed. +# This value must either be `/` or not end with a slash (e.g. `/go-neb`). +matrix_bot_go_neb_path_prefix: / + +matrix_bot_go_neb_base_url: "{{ matrix_bot_go_neb_scheme }}://{{ matrix_bot_go_neb_hostname }}{{ matrix_bot_go_neb_path_prefix }}{{ '' if matrix_bot_go_neb_path_prefix == '/' else '/' }}" matrix_bot_go_neb_base_path: "{{ matrix_base_data_path }}/go-neb" matrix_bot_go_neb_config_path: "{{ matrix_bot_go_neb_base_path }}/config" @@ -13,11 +24,63 @@ matrix_bot_go_neb_config_path_in_container: "/config/config.yaml" matrix_bot_go_neb_data_path: "{{ matrix_bot_go_neb_base_path }}/data" matrix_bot_go_neb_data_store_path: "{{ matrix_bot_go_neb_data_path }}/store" +matrix_bot_go_neb_container_image: "{{ matrix_bot_go_neb_container_image_registry_prefix }}matrixdotorg/go-neb:{{ matrix_bot_go_neb_container_image_tag }}" +matrix_bot_go_neb_container_image_tag: "{{ matrix_bot_go_neb_version }}" +matrix_bot_go_neb_container_image_force_pull: "{{ matrix_bot_go_neb_container_image.endswith(':latest') }}" +matrix_bot_go_neb_container_image_registry_prefix: docker.io/ + +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_bot_go_neb_container_network: matrix-bot-go-neb + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_bot_go_neb_container_additional_networks: [] + # Controls whether the matrix-bot-go-neb container exposes its HTTP port (tcp/4050 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:4050"), or empty string to not expose. matrix_bot_go_neb_container_http_host_bind_port: '' +# matrix_bot_go_neb_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_bot_go_neb_container_labels_additional_labels`. +matrix_bot_go_neb_container_labels_traefik_enabled: true +matrix_bot_go_neb_container_labels_traefik_docker_network: "{{ matrix_bot_go_neb_container_network }}" +matrix_bot_go_neb_container_labels_traefik_hostname: "{{ matrix_bot_go_neb_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/go-neb`). +matrix_bot_go_neb_container_labels_traefik_path_prefix: "{{ matrix_bot_go_neb_path_prefix }}" +matrix_bot_go_neb_container_labels_traefik_rule: "Host(`{{ matrix_bot_go_neb_container_labels_traefik_hostname }}`){% if matrix_bot_go_neb_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_bot_go_neb_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_bot_go_neb_container_labels_traefik_priority: 0 +matrix_bot_go_neb_container_labels_traefik_entrypoints: web-secure +matrix_bot_go_neb_container_labels_traefik_tls: "{{ matrix_bot_go_neb_container_labels_traefik_entrypoints != 'web' }}" +matrix_bot_go_neb_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_bot_go_neb_container_labels_traefik_additional_response_headers_custom` +matrix_bot_go_neb_container_labels_traefik_additional_response_headers: "{{ matrix_bot_go_neb_container_labels_traefik_additional_response_headers_auto | combine(matrix_bot_go_neb_container_labels_traefik_additional_response_headers_custom) }}" +matrix_bot_go_neb_container_labels_traefik_additional_response_headers_auto: | + {{ + {} + | combine ({'X-XSS-Protection': matrix_bot_go_neb_http_header_xss_protection} if matrix_bot_go_neb_http_header_xss_protection else {}) + | combine ({'X-Frame-Options': matrix_bot_go_neb_http_header_frame_options} if matrix_bot_go_neb_http_header_frame_options else {}) + | combine ({'X-Content-Type-Options': matrix_bot_go_neb_http_header_content_type_options} if matrix_bot_go_neb_http_header_content_type_options else {}) + | combine ({'Content-Security-Policy': matrix_bot_go_neb_http_header_content_security_policy} if matrix_bot_go_neb_http_header_content_security_policy else {}) + | combine ({'Permission-Policy': matrix_bot_go_neb_http_header_content_permission_policy} if matrix_bot_go_neb_http_header_content_permission_policy else {}) + | combine ({'Strict-Transport-Security': matrix_bot_go_neb_http_header_strict_transport_security} if matrix_bot_go_neb_http_header_strict_transport_security and matrix_bot_go_neb_container_labels_traefik_tls else {}) + }} +matrix_bot_go_neb_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_bot_go_neb_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_bot_go_neb_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_bot_go_neb_container_labels_additional_labels: '' + # A list of extra arguments to pass to the container matrix_bot_go_neb_container_extra_arguments: [] @@ -27,6 +90,57 @@ matrix_bot_go_neb_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-bot-go-neb.service wants matrix_bot_go_neb_systemd_wanted_services_list: [] +# Specifies the value of the `X-XSS-Protection` header +# Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. +# +# Learn more about it is here: +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection +# - https://portswigger.net/web-security/cross-site-scripting/reflected +matrix_bot_go_neb_http_header_xss_protection: "1; mode=block" + +# Specifies the value of the `X-Frame-Options` header which controls whether framing can happen. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options +matrix_bot_go_neb_http_header_frame_options: SAMEORIGIN + +# Specifies the value of the `X-Content-Type-Options` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options +matrix_bot_go_neb_http_header_content_type_options: nosniff + +# Specifies the value of the `Content-Security-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy +matrix_bot_go_neb_http_header_content_security_policy: frame-ancestors 'self' + +# Specifies the value of the `Permission-Policy` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permission-Policy +matrix_bot_go_neb_http_header_content_permission_policy: "{{ 'interest-cohort=()' if matrix_bot_go_neb_floc_optout_enabled else '' }}" + +# Specifies the value of the `Strict-Transport-Security` header. +# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +matrix_bot_go_neb_http_header_strict_transport_security: "max-age=31536000; includeSubDomains{{ '; preload' if matrix_bot_go_neb_hsts_preload_enabled else '' }}" + +# Controls whether to send a "Permissions-Policy interest-cohort=();" header along with all responses +# +# Learn more about what it is here: +# - https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea +# - https://paramdeo.com/blog/opting-your-website-out-of-googles-floc-network +# - https://amifloced.org/ +# +# Of course, a better solution is to just stop using browsers (like Chrome), which participate in such tracking practices. +# See: `matrix_bot_go_neb_content_permission_policy` +matrix_bot_go_neb_floc_optout_enabled: true + +# Controls if HSTS preloading is enabled +# +# In its strongest and recommended form, the [HSTS policy](https://www.chromium.org/hsts) includes all subdomains, and +# indicates a willingness to be "preloaded" into browsers: +# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload` +# For more information visit: +# - https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security +# - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security +# - https://hstspreload.org/#opt-in +# See: `matrix_bot_go_neb_http_header_strict_transport_security` +matrix_bot_go_neb_hsts_preload_enabled: false + # Database-related configuration fields. # # MUST be "sqlite3". No other type is supported. diff --git a/roles/custom/matrix-bot-go-neb/tasks/setup_install.yml b/roles/custom/matrix-bot-go-neb/tasks/install.yml similarity index 63% rename from roles/custom/matrix-bot-go-neb/tasks/setup_install.yml rename to roles/custom/matrix-bot-go-neb/tasks/install.yml index 1db16b91..9cb85db3 100644 --- a/roles/custom/matrix-bot-go-neb/tasks/setup_install.yml +++ b/roles/custom/matrix-bot-go-neb/tasks/install.yml @@ -1,8 +1,5 @@ --- -- ansible.builtin.set_fact: - matrix_bot_go_neb_requires_restart: false - - name: Ensure go-neb paths exist ansible.builtin.file: path: "{{ item.path }}" @@ -16,17 +13,6 @@ - {path: "{{ matrix_bot_go_neb_data_store_path }}", when: true} when: "item.when | bool" -- name: Ensure go-neb image is pulled - community.docker.docker_image: - name: "{{ matrix_bot_go_neb_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_bot_go_neb_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_go_neb_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - - name: Ensure go-neb config installed ansible.builtin.copy: content: "{{ matrix_bot_go_neb_configuration | to_nice_yaml(indent=2, width=999999) }}" @@ -35,16 +21,35 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" +- name: Ensure go-neb support files installed + ansible.builtin.template: + src: "{{ role_path }}/templates/{{ item }}.j2" + dest: "{{ matrix_bot_go_neb_base_path }}/{{ item }}" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - env + - labels + +- name: Ensure go-neb container network is created + community.general.docker_network: + name: "{{ matrix_bot_go_neb_container_network }}" + driver: bridge + +- name: Ensure go-neb container image is pulled + community.docker.docker_image: + name: "{{ matrix_bot_go_neb_container_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_bot_go_neb_container_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_go_neb_container_image_force_pull }}" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + - name: Ensure matrix-bot-go-neb.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-bot-go-neb.service.j2" dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-go-neb.service" mode: 0644 - register: matrix_bot_go_neb_systemd_service_result - -- name: Ensure matrix-bot-go-neb.service restarted, if necessary - ansible.builtin.service: - name: "matrix-bot-go-neb.service" - state: restarted - daemon_reload: true - when: "matrix_bot_go_neb_requires_restart | bool" diff --git a/roles/custom/matrix-bot-go-neb/tasks/main.yml b/roles/custom/matrix-bot-go-neb/tasks/main.yml index 126f6ae0..3249fc7b 100644 --- a/roles/custom/matrix-bot-go-neb/tasks/main.yml +++ b/roles/custom/matrix-bot-go-neb/tasks/main.yml @@ -4,7 +4,7 @@ - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" when: "run_setup | bool and matrix_bot_go_neb_enabled | bool" - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" when: "run_setup | bool and matrix_bot_go_neb_enabled | bool" tags: - setup-all @@ -13,7 +13,7 @@ - install-bot-go-neb - block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" when: "run_setup | bool and not matrix_bot_go_neb_enabled | bool" tags: - setup-all diff --git a/roles/custom/matrix-bot-go-neb/tasks/setup_uninstall.yml b/roles/custom/matrix-bot-go-neb/tasks/uninstall.yml similarity index 100% rename from roles/custom/matrix-bot-go-neb/tasks/setup_uninstall.yml rename to roles/custom/matrix-bot-go-neb/tasks/uninstall.yml diff --git a/roles/custom/matrix-bot-go-neb/templates/env.j2 b/roles/custom/matrix-bot-go-neb/templates/env.j2 new file mode 100644 index 00000000..22e605e5 --- /dev/null +++ b/roles/custom/matrix-bot-go-neb/templates/env.j2 @@ -0,0 +1,8 @@ +BIND_ADDRESS=:4050 + +DATABASE_TYPE={{ matrix_bot_go_neb_database_engine }} +DATABASE_URL={{ matrix_bot_go_neb_storage_database }} + +BASE_URL={{ matrix_bot_go_neb_base_url }} + +CONFIG_FILE={{ matrix_bot_go_neb_config_path_in_container }} diff --git a/roles/custom/matrix-bot-go-neb/templates/labels.j2 b/roles/custom/matrix-bot-go-neb/templates/labels.j2 new file mode 100644 index 00000000..0ad8d98c --- /dev/null +++ b/roles/custom/matrix-bot-go-neb/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_bot_go_neb_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_bot_go_neb_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_bot_go_neb_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_bot_go_neb_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-slashless-redirect.redirectregex.regex=({{ matrix_bot_go_neb_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + [matrix_bot_go_neb_identifier + '-slashless-redirect'] %} +{% endif %} + +{% if matrix_bot_go_neb_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-strip-prefix.stripprefix.prefixes={{ matrix_bot_go_neb_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + [matrix_bot_go_neb_identifier + '-strip-prefix'] %} +{% endif %} + +{% if matrix_bot_go_neb_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_bot_go_neb_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + [matrix_bot_go_neb_identifier + '-add-headers'] %} +{% endif %} + +traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.rule={{ matrix_bot_go_neb_container_labels_traefik_rule }} +{% if matrix_bot_go_neb_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.priority={{ matrix_bot_go_neb_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.service={{ matrix_bot_go_neb_identifier }} +{% if middlewares | length > 0 %} +traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.entrypoints={{ matrix_bot_go_neb_container_labels_traefik_entrypoints }} +traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.tls={{ matrix_bot_go_neb_container_labels_traefik_tls | to_json }} +{% if matrix_bot_go_neb_container_labels_traefik_tls %} +traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.tls.certResolver={{ matrix_bot_go_neb_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.{{ matrix_bot_go_neb_identifier }}.loadbalancer.server.port=4050 +{% endif %} + +{{ matrix_bot_go_neb_container_labels_additional_labels }} diff --git a/roles/custom/matrix-bot-go-neb/templates/systemd/matrix-bot-go-neb.service.j2 b/roles/custom/matrix-bot-go-neb/templates/systemd/matrix-bot-go-neb.service.j2 index a57df57e..150cd665 100644 --- a/roles/custom/matrix-bot-go-neb/templates/systemd/matrix-bot-go-neb.service.j2 +++ b/roles/custom/matrix-bot-go-neb/templates/systemd/matrix-bot-go-neb.service.j2 @@ -16,31 +16,37 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-go-neb 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-go-neb 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-bot-go-neb \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-bot-go-neb \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_bot_go_neb_container_network }} \ {% if matrix_bot_go_neb_container_http_host_bind_port %} -p {{ matrix_bot_go_neb_container_http_host_bind_port }}:4050 \ {% endif %} - -e 'BIND_ADDRESS=:4050' \ - -e 'DATABASE_TYPE={{ matrix_bot_go_neb_database_engine }}' \ - -e 'BASE_URL=https://{{ matrix_server_fqn_bot_go_neb }}' \ - -e 'CONFIG_FILE={{ matrix_bot_go_neb_config_path_in_container }}' \ - -e 'DATABASE_URL={{ matrix_bot_go_neb_storage_database }}' \ + --env-file={{ matrix_bot_go_neb_base_path }}/env \ + --label-file={{ matrix_bot_go_neb_base_path }}/labels \ --mount type=bind,src={{ matrix_bot_go_neb_config_path }},dst=/config,ro \ --mount type=bind,src={{ matrix_bot_go_neb_data_path }},dst=/data \ --entrypoint=/bin/sh \ {% for arg in matrix_bot_go_neb_container_extra_arguments %} {{ arg }} \ {% endfor %} - {{ matrix_bot_go_neb_docker_image }} \ + {{ matrix_bot_go_neb_container_image }} \ -c "go-neb /config/config.yaml" +{% for network in matrix_bot_go_neb_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-bot-go-neb +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-bot-go-neb + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-go-neb 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-go-neb 2>/dev/null || true' + Restart=always RestartSec=30 SyslogIdentifier=matrix-bot-go-neb diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index daabcdb9..f7f214f3 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -63,11 +63,6 @@ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_rule: "Host( matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint: "{{ matrix_federation_traefik_entrypoint }}" matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoints: "{{ matrix_nginx_proxy_container_labels_traefik_proxy_matrix_federation_entrypoint }}" -matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled: false -matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_hostname: "{{ matrix_server_fqn_bot_go_neb }}" -matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" -matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_rule: "Host(`{{ matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_hostname }}`)" - matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled: false matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_hostname: "{{ matrix_server_fqn_jitsi }}" matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_tls: "{{ matrix_nginx_proxy_container_labels_traefik_entrypoints != 'web' }}" diff --git a/roles/custom/matrix-nginx-proxy/templates/labels.j2 b/roles/custom/matrix-nginx-proxy/templates/labels.j2 index 4cf78b2e..4fb567da 100644 --- a/roles/custom/matrix-nginx-proxy/templates/labels.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/labels.j2 @@ -39,18 +39,6 @@ traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints={{ matrix_ {% endif %} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_enabled %} -# Go NEB bot -traefik.http.routers.matrix-nginx-proxy-bot_go_neb.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_rule }} -traefik.http.routers.matrix-nginx-proxy-bot_go_neb.service=matrix-nginx-proxy-web -traefik.http.routers.matrix-nginx-proxy-bot_go_neb.tls={{ matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_tls | to_json }} -{% if matrix_nginx_proxy_container_labels_traefik_proxy_bot_go_neb_tls %} -traefik.http.routers.matrix-nginx-proxy-bot_go_neb.tls.certResolver={{ matrix_nginx_proxy_container_labels_traefik_tls_certResolver }} -{% endif %} -traefik.http.routers.matrix-nginx-proxy-bot_go_neb.entrypoints={{ matrix_nginx_proxy_container_labels_traefik_entrypoints }} -{% endif %} - - {% if matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_enabled %} # Jitsi traefik.http.routers.matrix-nginx-proxy-jitsi.rule={{ matrix_nginx_proxy_container_labels_traefik_proxy_jitsi_rule }} From 6a6761cb88afe14d26c9458ea98f15cf662bfa91 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 5 Mar 2023 10:17:07 +0000 Subject: [PATCH 292/382] Update ntfy 2.1.1 -> 2.1.2 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 3dd08913..158b4fde 100644 --- a/requirements.yml +++ b/requirements.yml @@ -43,7 +43,7 @@ version: v9.4.3-0 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git - version: v2.1.1-0 + version: v2.1.2-0 - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.9-0 From 493de7316c2b35b40e7326ef3843803ffdb5ab08 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sun, 5 Mar 2023 14:52:53 +0000 Subject: [PATCH 293/382] update honoroit 0.9.16 -> 0.9.17 * healthchecks.io integration * mutex on forwarding messages into thread * fix in prefixes handling * send error messages as thread reply when possible --- roles/custom/matrix-bot-honoroit/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-honoroit/defaults/main.yml b/roles/custom/matrix-bot-honoroit/defaults/main.yml index e880df4f..42f76682 100644 --- a/roles/custom/matrix-bot-honoroit/defaults/main.yml +++ b/roles/custom/matrix-bot-honoroit/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_honoroit_docker_repo: "https://gitlab.com/etke.cc/honoroit.git" matrix_bot_honoroit_docker_repo_version: "{{ matrix_bot_honoroit_version }}" matrix_bot_honoroit_docker_src_files_path: "{{ matrix_base_data_path }}/honoroit/docker-src" -matrix_bot_honoroit_version: v0.9.16 +matrix_bot_honoroit_version: v0.9.17 matrix_bot_honoroit_docker_image: "{{ matrix_bot_honoroit_docker_image_name_prefix }}etke.cc/honoroit:{{ matrix_bot_honoroit_version }}" matrix_bot_honoroit_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_honoroit_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_honoroit_docker_image_force_pull: "{{ matrix_bot_honoroit_docker_image.endswith(':latest') }}" From 449b51588e32a165ccf7e95c14f0370b0edd2b60 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 08:51:21 +0200 Subject: [PATCH 294/382] Remove usage of undefined matrix_bot_go_neb_identifier variable This is a mistake made in 10b53503704d192d29008745. Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2540 --- .../matrix-bot-go-neb/templates/labels.j2 | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/roles/custom/matrix-bot-go-neb/templates/labels.j2 b/roles/custom/matrix-bot-go-neb/templates/labels.j2 index 0ad8d98c..54aa2583 100644 --- a/roles/custom/matrix-bot-go-neb/templates/labels.j2 +++ b/roles/custom/matrix-bot-go-neb/templates/labels.j2 @@ -8,38 +8,38 @@ traefik.docker.network={{ matrix_bot_go_neb_container_labels_traefik_docker_netw {% set middlewares = [] %} {% if matrix_bot_go_neb_container_labels_traefik_path_prefix != '/' %} -traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-slashless-redirect.redirectregex.regex=({{ matrix_bot_go_neb_container_labels_traefik_path_prefix | quote }})$ -traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-slashless-redirect.redirectregex.replacement=${1}/ -{% set middlewares = middlewares + [matrix_bot_go_neb_identifier + '-slashless-redirect'] %} +traefik.http.middlewares.matrix-bot-go-neb-slashless-redirect.redirectregex.regex=({{ matrix_bot_go_neb_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-bot-go-neb-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-bot-go-neb-slashless-redirect'] %} {% endif %} {% if matrix_bot_go_neb_container_labels_traefik_path_prefix != '/' %} -traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-strip-prefix.stripprefix.prefixes={{ matrix_bot_go_neb_container_labels_traefik_path_prefix }} -{% set middlewares = middlewares + [matrix_bot_go_neb_identifier + '-strip-prefix'] %} +traefik.http.middlewares.matrix-bot-go-neb-strip-prefix.stripprefix.prefixes={{ matrix_bot_go_neb_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-bot-go-neb-strip-prefix'] %} {% endif %} {% if matrix_bot_go_neb_container_labels_traefik_additional_response_headers.keys() | length > 0 %} {% for name, value in matrix_bot_go_neb_container_labels_traefik_additional_response_headers.items() %} -traefik.http.middlewares.{{ matrix_bot_go_neb_identifier }}-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +traefik.http.middlewares.matrix-bot-go-neb-add-headers.headers.customresponseheaders.{{ name }}={{ value }} {% endfor %} -{% set middlewares = middlewares + [matrix_bot_go_neb_identifier + '-add-headers'] %} +{% set middlewares = middlewares + ['matrix-bot-go-neb-add-headers'] %} {% endif %} -traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.rule={{ matrix_bot_go_neb_container_labels_traefik_rule }} +traefik.http.routers.matrix-bot-go-neb.rule={{ matrix_bot_go_neb_container_labels_traefik_rule }} {% if matrix_bot_go_neb_container_labels_traefik_priority | int > 0 %} -traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.priority={{ matrix_bot_go_neb_container_labels_traefik_priority }} +traefik.http.routers.matrix-bot-go-neb.priority={{ matrix_bot_go_neb_container_labels_traefik_priority }} {% endif %} -traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.service={{ matrix_bot_go_neb_identifier }} +traefik.http.routers.matrix-bot-go-neb.service=matrix-bot-go-neb {% if middlewares | length > 0 %} -traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.middlewares={{ middlewares | join(',') }} +traefik.http.routers.matrix-bot-go-neb.middlewares={{ middlewares | join(',') }} {% endif %} -traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.entrypoints={{ matrix_bot_go_neb_container_labels_traefik_entrypoints }} -traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.tls={{ matrix_bot_go_neb_container_labels_traefik_tls | to_json }} +traefik.http.routers.matrix-bot-go-neb.entrypoints={{ matrix_bot_go_neb_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-bot-go-neb.tls={{ matrix_bot_go_neb_container_labels_traefik_tls | to_json }} {% if matrix_bot_go_neb_container_labels_traefik_tls %} -traefik.http.routers.{{ matrix_bot_go_neb_identifier }}.tls.certResolver={{ matrix_bot_go_neb_container_labels_traefik_tls_certResolver }} +traefik.http.routers.matrix-bot-go-neb.tls.certResolver={{ matrix_bot_go_neb_container_labels_traefik_tls_certResolver }} {% endif %} -traefik.http.services.{{ matrix_bot_go_neb_identifier }}.loadbalancer.server.port=4050 +traefik.http.services.matrix-bot-go-neb.loadbalancer.server.port=4050 {% endif %} {{ matrix_bot_go_neb_container_labels_additional_labels }} From bf2b54080789f7e82eeeb118f1ddccbc7ffffb83 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 09:08:04 +0200 Subject: [PATCH 295/382] Harden Traefik security by accessing the Docker API through docker-socket-proxy With these changes, we: - install https://github.com/Tecnativa/docker-socket-proxy via the https://github.com/devture/com.devture.ansible.role.container_socket_proxy Ansible role - make Traefik access the Docker API via TCP by connecting to this socket proxy - .. which allows us to run the Traefik container with less privileges (non-`root`, dropped capabilities) --- group_vars/matrix_servers | 43 ++++++++++++++++++++++++++++++++++++++- playbooks/matrix.yml | 2 ++ requirements.yml | 5 ++++- 3 files changed, 48 insertions(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 0a3ff00b..d4bcd027 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -356,7 +356,9 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-user-verification-service.service', 'priority': 800, 'groups': ['matrix', 'matrix-user-verification-service']}] if matrix_user_verification_service_enabled else []) + - ([{'name': 'devture-traefik.service', 'priority': 3000, 'groups': ['matrix', 'traefik', 'reverse-proxies']}] if devture_traefik_enabled else []) + ([{'name': (devture_container_socket_proxy_identifier + '.service'), 'priority': 2900, 'groups': ['matrix', 'reverse-proxies', 'container-socket-proxy']}] if devture_container_socket_proxy_enabled else []) + + + ([{'name': (devture_traefik_identifier + '.service'), 'priority': 3000, 'groups': ['matrix', 'traefik', 'reverse-proxies']}] if devture_traefik_enabled else []) + ([{'name': (devture_traefik_certs_dumper_identifier + '.service'), 'priority': 3500, 'groups': ['matrix', 'traefik-certs-dumper']}] if devture_traefik_certs_dumper_enabled else []) }} @@ -3821,6 +3823,31 @@ matrix_user_verification_service_uvs_auth_token: "{{ '%s' | format(matrix_homese ###################################################################### +######################################################################## +# # +# com.devture.ansible.role.container_socket_proxy # +# # +######################################################################## + +devture_container_socket_proxy_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' }}" + +devture_container_socket_proxy_identifier: matrix-container-socket-proxy + +devture_container_socket_proxy_base_path: "{{ matrix_base_data_path }}/container-socket-proxy" + +devture_container_socket_proxy_uid: "{{ matrix_user_uid }}" +devture_container_socket_proxy_gid: "{{ matrix_user_gid }}" + +# Traefik requires read access to the containers APIs to do its job +devture_container_socket_proxy_api_containers_enabled: true + +######################################################################## +# # +# /com.devture.ansible.role.container_socket_proxy # +# # +######################################################################## + + ######################################################################## # # # com.devture.ansible.role.traefik # @@ -3842,6 +3869,20 @@ devture_traefik_additional_entrypoints_auto: devture_traefik_additional_domains_to_obtain_certificates_for: "{{ matrix_ssl_additional_domains_to_obtain_certificates_for }}" +devture_traefik_config_providers_docker_endpoint: "{{ devture_container_socket_proxy_endpoint if devture_container_socket_proxy_enabled else 'unix:///var/run/docker.sock' }}" + +devture_traefik_container_additional_networks: | + {{ + ([devture_container_socket_proxy_container_network] if devture_container_socket_proxy_enabled else []) + }} + +devture_traefik_systemd_required_services_list: | + {{ + (['docker.service']) + + + ([devture_container_socket_proxy_identifier + '.service'] if devture_container_socket_proxy_enabled else []) + }} + ######################################################################## # # # /com.devture.ansible.role.traefik # diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 6a66dc58..6e348df2 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -119,6 +119,8 @@ - custom/matrix-user-creator - custom/matrix-common-after + - role: galaxy/com.devture.ansible.role.container_socket_proxy + - when: matrix_playbook_traefik_role_enabled | bool role: galaxy/com.devture.ansible.role.traefik diff --git a/requirements.yml b/requirements.yml index 158b4fde..ad1f5eac 100644 --- a/requirements.yml +++ b/requirements.yml @@ -51,8 +51,11 @@ - src: git+https://gitlab.com/etke.cc/roles/etherpad.git version: v1.8.18-2 +- src: git+https://github.com/devture/com.devture.ansible.role.container_socket_proxy.git + version: v0.1.1-0 + - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: v2.9.8-0 + version: v2.9.8-1 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: v2.8.1-0 From adcc6d9723086f65f1a7284a4d3eee03de56ac22 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 09:32:13 +0200 Subject: [PATCH 296/382] Relocate Traefik (to matrix-traefik.service && /matrix/traefik base path) The migration is automatic. Existing users should experience a bit of downtime until the playbook runs to completion, but don't need to do anything manually. This change is provoked by https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2535 While my statements there ("Traefik is a shared component among sibling/related playbooks and should retain its global non-matrix-prefixed name and path") do make sense, there's another point of view as well. With the addition of docker-socket-proxy support in bf2b54080789f7e, we potentially introduced another non-`matrix-`-prefixed systemd service and global path (`/devture-container-socket-proxy`). It would have started to become messy. Traefik always being called `devture-traefik.service` and using the `/devture-traefik` path has the following downsides: - different playbooks may write to the same place, unintentionally, before you disable the Traefik role in some of them. If each playbook manages its own installation, no such conflicts arise and you'll learn about the conflict when one of them starts its Traefik service and fails because the ports are already in use - the data is scattered - backing up `/matrix` is no longer enough when some stuff lives in `/devture-traefik` or `/devture-container-socket-proxy` as well; similarly, deleting `/matrix` is no longer enough to clean up For this reason, the Traefik instance managed by this playbook will now be called `matrix-traefik` and live under `/matrix/traefik`. This also makes it obvious to users running multiple playbooks, which Traefik instance (powered by which playbook) is the active one. Previously, you'd look at `devture-traefik.service` and wonder which role was managing it. --- group_vars/matrix_servers | 4 +++ .../matrix-base/templates/bin/remove-all.j2 | 11 +++--- .../devture_traefik_to_matrix_traefik.yml | 35 +++++++++++++++++++ .../matrix_playbook_migration/tasks/main.yml | 9 +++++ 4 files changed, 52 insertions(+), 7 deletions(-) create mode 100644 roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index d4bcd027..c9f78345 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3858,6 +3858,10 @@ devture_container_socket_proxy_api_containers_enabled: true # See the comment there for more details about why we have both `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`. devture_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' }}" +devture_traefik_identifier: matrix-traefik + +devture_traefik_base_path: "{{ matrix_base_data_path }}/traefik" + devture_traefik_uid: "{{ matrix_user_uid }}" devture_traefik_gid: "{{ matrix_user_gid }}" diff --git a/roles/custom/matrix-base/templates/bin/remove-all.j2 b/roles/custom/matrix-base/templates/bin/remove-all.j2 index 46e1f757..2733ad78 100644 --- a/roles/custom/matrix-base/templates/bin/remove-all.j2 +++ b/roles/custom/matrix-base/templates/bin/remove-all.j2 @@ -21,19 +21,16 @@ else rm -f {{ devture_systemd_docker_base_systemd_path }}/$s done - echo "Stop and remove devture services" - - for s in $(find {{ devture_systemd_docker_base_systemd_path }}/ -type f -name "devture-*" -printf "%f\n"); do - systemctl disable --now $s - rm -f {{ devture_systemd_docker_base_systemd_path }}/$s - done - systemctl daemon-reload echo "Remove unused Docker images and resources" docker system prune -af + echo "Remove Docker matrix network (should be gone already, but ..)" docker network rm {{ matrix_docker_network }} + + echo "This playbook creates a lot of matrix-* networks. Consider removing them manually ('docker network ls', followed by 'docker network rm NETWORK_NAME')" + echo "Remove {{ matrix_base_data_path }} directory" rm -fr "{{ matrix_base_data_path }}" exit 0 diff --git a/roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml b/roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml new file mode 100644 index 00000000..7d1b1190 --- /dev/null +++ b/roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml @@ -0,0 +1,35 @@ +--- + +# This migrates Traefik from the old path (`/devture-traefik`) to the new path (`/matrix/traefik`, controlled by `devture_traefik_base_path`), +# and from the old hardcoded systemd service name (`devture-traefik.service`) to the new one (`matrix-traefik.service`, controlled by `devture_traefik_identifier`). +# +# Here, we merely disable (and stop) the old systemd service and relocate the data (`/devture-traefik` directory). +# The Traefik role itself (running later) will then ensure this data is up-to-date and will set up the new systemd service. + +# It only makes sense to migrate if the identifier or path are different than the default (what we were using before). +- when: "devture_traefik_identifier != 'devture-postgres' or devture_traefik_base_path != '/devture-traefik'" + block: + - name: Check existence of devture-traefik.service systemd service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/devture-traefik.service" + register: devture_traefik_service_stat + + - when: devture_traefik_service_stat.stat.exists | bool + block: + - name: Ensure devture-traefik.service systemd service is stopped + ansible.builtin.systemd: + name: devture-traefik + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure Traefik systemd service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/devture-traefik.service" + state: absent + + - name: Ensure Traefik directory relocated + ansible.builtin.command: + cmd: "mv /devture-traefik {{ devture_traefik_base_path }}" + creates: "{{ devture_traefik_base_path }}" + removes: "/devture-traefik" diff --git a/roles/custom/matrix_playbook_migration/tasks/main.yml b/roles/custom/matrix_playbook_migration/tasks/main.yml index 96b68b60..e2b29384 100644 --- a/roles/custom/matrix_playbook_migration/tasks/main.yml +++ b/roles/custom/matrix_playbook_migration/tasks/main.yml @@ -11,3 +11,12 @@ tags: - setup-all - install-all + +- when: matrix_playbook_traefik_role_enabled | bool + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/devture_traefik_to_matrix_traefik.yml" + tags: + - setup-all + - install-all + - setup-traefik + - install-traefik From 30f10347670d2fcc3fafc3ac81659dcb6d909cdc Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 09:51:14 +0200 Subject: [PATCH 297/382] Remove matrix_playbook_traefik_role_enabled variable and devture-traefik references The variable was necessary when multiple playbooks could have potentially tried to manage a shared `devture-traefik.serivce` systemd service and shared `/devture-traefik` directory. Since adcc6d9723086f65f1a72, we use our own `/matrix/traefik` (`matrix-traefik.service`) installation and no conflicts can arise. It's safe to always enable the role, just like we do with all the other roles. --- CHANGELOG.md | 4 ++-- docs/configuring-playbook-ssl-certificates.md | 4 ++-- group_vars/matrix_servers | 20 ++----------------- playbooks/matrix.yml | 3 +-- roles/custom/matrix-base/defaults/main.yml | 2 +- .../matrix_playbook_migration/tasks/main.yml | 2 +- .../tasks/validate_config.yml | 1 + 7 files changed, 10 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 790406f3..1aedacc3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -223,9 +223,9 @@ devture_traefik_config_certificatesResolvers_acme_email: YOUR_EMAIL_ADDRESS You may still need to keep certain old `matrix_nginx_proxy_*` variables (like `matrix_nginx_proxy_base_domain_serving_enabled`), even when using Traefik. For now, we recommend keeping all `matrix_nginx_proxy_*` variables just in case. In the future, reliance on `matrix-nginx-proxy` will be removed. -Switching to Traefik will obtain new SSL certificates from Let's Encrypt (stored in `/devture-traefik/ssl/acme.json`). **The switch is reversible**. You can always go back to `playbook-managed-nginx` if Traefik is causing you trouble. +Switching to Traefik will obtain new SSL certificates from Let's Encrypt (stored in `/matrix/traefik/ssl/acme.json`). **The switch is reversible**. You can always go back to `playbook-managed-nginx` if Traefik is causing you trouble. -**Note**: toggling `matrix_playbook_reverse_proxy_type` between Traefik and nginx will uninstall the Traefik role and all of its data (under `/devture-traefik`), so you may run into a Let's Encrypt rate limit if you do it often. +**Note**: toggling `matrix_playbook_reverse_proxy_type` between Traefik and nginx will uninstall the Traefik role and all of its data (under `/matrix/traefik`), so you may run into a Let's Encrypt rate limit if you do it often. Treafik directly reverse-proxies to **some** services right now, but for most other services it goes through `matrix-nginx-proxy` (e.g. Traefik -> `matrix-nginx-proxy` -> [Ntfy](docs/configuring-playbook-ntfy.md)). So, even if you opt into Traefik, you'll still see `matrix-nginx-proxy` being installed in local-only mode. This will improve with time. diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index 13d5f609..9fa9f84b 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -54,7 +54,7 @@ devture_traefik_ssl_dir_enabled: true # Tell Traefik to load our custom configuration file (certificates.yml). # The file is created below, in `matrix_aux_file_definitions`. -# The `/config/..` path is an in-container path, not a path on the host (like `/devture-traefik/config`). Do not change it! +# The `/config/..` path is an in-container path, not a path on the host (like `/matrix/traefik/config`). Do not change it! devture_traefik_configuration_extension_yaml: | providers: file: @@ -85,7 +85,7 @@ matrix_aux_file_definitions: # HERE # Create the custom Traefik configuration. - # The `/ssl/..` paths below are in-container paths, not paths on the host (/`devture-traefik/ssl/..`). Do not change them! + # The `/ssl/..` paths below are in-container paths, not paths on the host (/`matrix/traefik/ssl/..`). Do not change them! - dest: "{{ devture_traefik_config_dir_path }}/certificates.yml" content: | tls: diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index c9f78345..5c4c1bdd 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -19,23 +19,9 @@ # Also see `devture_docker_sdk_for_python_installation_enabled`. matrix_playbook_docker_installation_enabled: true -# Controls whether to run the Traefik role or not -# See the `com.devture.ansible.role.traefik` section below for role configuration. -# -# There's a difference between `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`. -# `devture_traefik_enabled` controls what the Traefik role would do - when not enabled, it will run uninstall tasks, etc. -# `matrix_playbook_traefik_role_enabled` controls if the Traefik role would even run at all. -# -# Sometimes, you're installing Traefik via a different (related playbook) which uses the same role. -# In such cases, you'd like to disable the role in this playbook from bothering with Traefik at all (`matrix_playbook_traefik_role_enabled: false`). -# If you used `devture_traefik_enabled: false` + `matrix_playbook_traefik_role_enabled: true` instead, you'd see the Treafik role here -# try to delete Traefik data (`/devture-traefik`) installed by the other playbook. -matrix_playbook_traefik_role_enabled: "{{ matrix_playbook_reverse_proxy_type != 'other-traefik-container' }}" - # Controls whether to attach Traefik labels to services. -# This is separate from `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`, -# because you may wish to disable Traefik installation by the playbook, yet still use Traefik -# installed in another way. +# This is separate from `devture_traefik_enabled`, because you may wish to disable Traefik installation by the playbook, +# yet still use Traefik installed in another way. matrix_playbook_traefik_labels_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" # Controls the additional network that reverse-proxyable services will be connected to. @@ -3854,8 +3840,6 @@ devture_container_socket_proxy_api_containers_enabled: true # # ######################################################################## -# To completely disable the Traefik role from running, use `matrix_playbook_traefik_role_enabled: false`. -# See the comment there for more details about why we have both `devture_traefik_enabled` and `matrix_playbook_traefik_role_enabled`. devture_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' }}" devture_traefik_identifier: matrix-traefik diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 6e348df2..e72269ee 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -121,8 +121,7 @@ - role: galaxy/com.devture.ansible.role.container_socket_proxy - - when: matrix_playbook_traefik_role_enabled | bool - role: galaxy/com.devture.ansible.role.traefik + - role: galaxy/com.devture.ansible.role.traefik - role: galaxy/com.devture.ansible.role.traefik_certs_dumper diff --git a/roles/custom/matrix-base/defaults/main.yml b/roles/custom/matrix-base/defaults/main.yml index 98a2167e..5e743925 100644 --- a/roles/custom/matrix-base/defaults/main.yml +++ b/roles/custom/matrix-base/defaults/main.yml @@ -284,7 +284,7 @@ matrix_homeserver_app_service_config_files_auto: [] # Valid options and a description of their behavior: # # - `playbook-managed-traefik` -# - the playbook will install devture-traefik +# - the playbook will run a managed Traefik instance (matrix-traefik) # - Traefik will do SSL termination, unless you disable it (e.g. `devture_traefik_config_entrypoint_web_secure_enabled: false`) # - if SSL termination is enabled (as it is by default), you need to populate: `devture_traefik_config_certificatesResolvers_acme_email` # - it will also install matrix-nginx-proxy in local-only mode, while we migrate the rest of the services to a Traefik-native mode of working diff --git a/roles/custom/matrix_playbook_migration/tasks/main.yml b/roles/custom/matrix_playbook_migration/tasks/main.yml index e2b29384..4a39e2c7 100644 --- a/roles/custom/matrix_playbook_migration/tasks/main.yml +++ b/roles/custom/matrix_playbook_migration/tasks/main.yml @@ -12,7 +12,7 @@ - setup-all - install-all -- when: matrix_playbook_traefik_role_enabled | bool +- when: devture_traefik_enabled | bool block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/devture_traefik_to_matrix_traefik.yml" tags: diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 296f2daf..d8fe1683 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -37,6 +37,7 @@ - {'old': 'matrix_prometheus_node_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled'} - {'old': 'matrix_prometheus_postgres_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled'} - {'old': 'matrix_playbook_traefik_certs_dumper_role_enabled', 'new': 'devture_traefik_certs_dumper_enabled'} + - {'old': 'matrix_playbook_traefik_role_enabled', 'new': 'devture_traefik_enabled'} - name: (Deprecation) Catch and report matrix_postgres variables ansible.builtin.fail: From e21c44347f971319c204ca129c7d27714689bc26 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 10:28:44 +0200 Subject: [PATCH 298/382] Add Traefik Dashboard warning --- docs/configuring-playbook-traefik.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/configuring-playbook-traefik.md b/docs/configuring-playbook-traefik.md index 4f6a7335..0485742d 100644 --- a/docs/configuring-playbook-traefik.md +++ b/docs/configuring-playbook-traefik.md @@ -35,6 +35,8 @@ devture_traefik_dashboard_basicauth_user: YOUR_USERNAME_HERE devture_traefik_dashboard_basicauth_password: YOUR_PASSWORD_HERE ``` +**WARNING**: enabling the dashboard on a hostname you use for something else (like `matrix_server_fqn_matrix` in the configuration above) may cause conflicts. Enabling the Traefik Dashboard makes Traefik capture all `/dashboard` and `/api` requests and forward them to itself. If any of the services hosted on the same hostname requires any of these 2 URL prefixes, you will experience problems. So far, we're not aware of any playbook services which occupy these endpoints and are likely to cause conflicts. + ## Additional configuration Use the `devture_traefik_configuration_extension_yaml` variable provided by the Traefik Ansible role to override or inject additional settings, even when no dedicated variable exists. From 26fdae3797419f0df5065109adb53cfe74564559 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 10:29:58 +0200 Subject: [PATCH 299/382] Upgrade com.devture.ansible.role.container_socket_proxy --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index ad1f5eac..013a6c53 100644 --- a/requirements.yml +++ b/requirements.yml @@ -52,7 +52,7 @@ version: v1.8.18-2 - src: git+https://github.com/devture/com.devture.ansible.role.container_socket_proxy.git - version: v0.1.1-0 + version: v0.1.1-1 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: v2.9.8-1 From 6e3aab65942e2da274fd283cae094ffd988ba4f7 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 11:00:42 +0200 Subject: [PATCH 300/382] Make devture-traefik to matrix-traefik migration code a little safer --- .../tasks/devture_traefik_to_matrix_traefik.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml b/roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml index 7d1b1190..f192cf01 100644 --- a/roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml +++ b/roles/custom/matrix_playbook_migration/tasks/devture_traefik_to_matrix_traefik.yml @@ -23,13 +23,13 @@ enabled: false daemon_reload: true - - name: Ensure Traefik systemd service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/devture-traefik.service" - state: absent - - name: Ensure Traefik directory relocated ansible.builtin.command: cmd: "mv /devture-traefik {{ devture_traefik_base_path }}" creates: "{{ devture_traefik_base_path }}" removes: "/devture-traefik" + + - name: Ensure Traefik systemd service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/devture-traefik.service" + state: absent From db4070fc75d62aa296f55716e67a0561c2767cdb Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 13:48:31 +0200 Subject: [PATCH 301/382] Remove some nginx references --- docs/faq.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 25d02e7c..7b67a5bf 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -125,7 +125,7 @@ This is similar to the [EMnify/matrix-synapse-auto-deploy](https://github.com/EM - this one installs everything in a single directory (`/matrix` by default) and **doesn't "contaminate" your server** with files all over the place -- this one **doesn't necessarily take over** ports 80 and 443. By default, it sets up nginx for you there, but you can also [use your own webserver](configuring-playbook-own-webserver.md) +- this one **doesn't necessarily take over** ports 80 and 443. By default, it sets up [Traefik](https://doc.traefik.io/traefik/) for you there, but you can also [use your own webserver](configuring-playbook-own-webserver.md) - this one **runs everything in Docker containers**, so it's likely more predictable and less fragile (see [Docker images used by this playbook](container-images.md)) @@ -322,7 +322,7 @@ matrix_playbook_docker_installation_enabled: false ### I run another webserver on the same server where I wish to install Matrix. What now? -By default, we install a webserver for you (nginx), but you can also use [your own webserver](configuring-playbook-own-webserver.md). +By default, we install a webserver for you ([Traefik](https://doc.traefik.io/traefik/)), but you can also use [your own webserver](configuring-playbook-own-webserver.md). ### How is the effective configuration determined? @@ -461,15 +461,8 @@ After verifying that everything still works after the Postgres upgrade, you can ### How do I debug or force SSL certificate renewal? -SSL certificate renewal normally happens automatically via [systemd timers](https://wiki.archlinux.org/index.php/Systemd/Timers). +SSL certificates are managed automatically by the [Traefik](https://doc.traefik.io/traefik/) reverse-proxy server. -If you're having trouble with SSL certificate renewal, you can inspect the renewal logs using: +If you're having trouble with SSL certificate renewal, check the Traefik logs (`journalctl -fu matrix-traefik`). -- `journalctl -fu matrix-ssl-lets-encrypt-certificates-renew.service` -- *or* by looking at the log files in `/matrix/ssl/log/` - -To trigger renewal, run: `systemctl start matrix-ssl-lets-encrypt-certificates-renew.service`. You can then take a look at the logs again. - -If you're using the integrated webserver (`matrix-nginx-proxy`), you can reload it manually like this: `systemctl reload matrix-nginx-proxy`. Reloading also happens periodically via a systemd timer. - -If you're [using your own webserver](configuring-playbook-own-webserver.md) instead of the integrated one (`matrix-nginx-proxy`) you may also need to reload/restart it, to make it pick up the renewed SSL certificate files. +If you're [using your own webserver](configuring-playbook-own-webserver.md) instead of the integrated one (Traefik), you should investigate in another way. From 2a872e95fcc171a0bb96317b4a811762957600e3 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 6 Mar 2023 13:52:22 +0200 Subject: [PATCH 302/382] Mention external roles in the FAQ --- docs/faq.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 7b67a5bf..48c658d1 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -336,12 +336,14 @@ Configuration variables are defined in multiple places in this playbook and are ### What configuration variables are available? -You can discover the variables you can override in each role (`role/matrix*/defaults/main.yml`). +You can discover the variables you can override in each role (`roles/*/*/defaults/main.yml`). As described in [How is the effective configuration determined?](#how-is-the-effective-configuration-determined), these role-defaults may be overriden by values defined in `group_vars/matrix_servers`. Refer to both of these for inspiration. Still, as mentioned in [Configuring the playbook](configuring-playbook.md), you're only ever supposed to edit your own `inventory/host_vars/matrix.DOMAIN/vars.yml` file and nothing else inside the playbook (unless you're meaning to contribute new features). +**Note**: some of the roles (`roles/galaxy/*`) live in separate repositories and are only installed after your run `just roles` (or `make roles`). + ### I'd like to adjust some configuration which doesn't have a corresponding variable. How do I do it? The playbook doesn't aim to expose all configuration settings for all services using variables. @@ -352,7 +354,9 @@ See [What configuration variables are available?](#what-configuration-variables- Besides that, each role (component) aims to provide a `matrix_SOME_COMPONENT_configuration_extension_yaml` (or `matrix_SOME_COMPONENT_configuration_extension_json`) variable, which can be used to override the configuration. -Check each role's `role/matrix*/defaults/main.yml` for the corresponding variable and an example for how use it. +Check each role's `roles/*/*/defaults/main.yml` for the corresponding variable and an example for how use it. + +**Note**: some of the roles (`roles/galaxy/*`) live in separate repositories and are only installed after your run `just roles` (or `make roles`). ## Installation From 023fe3ea08593b6786c93d641bd0c0daf2e2618d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 7 Mar 2023 11:57:56 +0200 Subject: [PATCH 303/382] Add sliding-sync support This allows people to try out the new Element X clients, which need to run against the sliding-sync proxy (https://github.com/matrix-org/sliding-sync). Supersedes https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2515 The code is based on the existing PR (#2515), but heavily reworked. Major changes: - lots of internal refactoring and variable renaming - fixed self-building to support non-amd64 architectures - changed to talk to the homeserver locally, over the container network (not publicly) - no more matrix-nginx-proxy support due to complexity (see below) - no more `matrix_server_fqn_sliding_sync_proxy` in favor of `matrix_sliding_sync_hostname` and `matrix_sliding_sync_path_prefix` - runs on `matrix.DOMAIN/sliding-sync` by default, so it can tried easily without having to create new DNS records --- group_vars/matrix_servers | 67 ++++++++++++++ playbooks/matrix.yml | 1 + .../static-files/well-known/matrix-client.j2 | 5 + .../matrix-sliding-sync/defaults/main.yml | 91 +++++++++++++++++++ .../matrix-sliding-sync/tasks/install.yml | 69 ++++++++++++++ .../custom/matrix-sliding-sync/tasks/main.yml | 19 ++++ .../matrix-sliding-sync/tasks/uninstall.yml | 25 +++++ .../tasks/validate_config.yml | 11 +++ .../matrix-sliding-sync/templates/env.j2 | 4 + .../matrix-sliding-sync/templates/labels.j2 | 45 +++++++++ .../systemd/matrix-sliding-sync.service.j2 | 49 ++++++++++ .../custom/matrix-sliding-sync/vars/main.yml | 5 + 12 files changed, 391 insertions(+) create mode 100644 roles/custom/matrix-sliding-sync/defaults/main.yml create mode 100644 roles/custom/matrix-sliding-sync/tasks/install.yml create mode 100644 roles/custom/matrix-sliding-sync/tasks/main.yml create mode 100644 roles/custom/matrix-sliding-sync/tasks/uninstall.yml create mode 100644 roles/custom/matrix-sliding-sync/tasks/validate_config.yml create mode 100644 roles/custom/matrix-sliding-sync/templates/env.j2 create mode 100644 roles/custom/matrix-sliding-sync/templates/labels.j2 create mode 100644 roles/custom/matrix-sliding-sync/templates/systemd/matrix-sliding-sync.service.j2 create mode 100644 roles/custom/matrix-sliding-sync/vars/main.yml diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 5c4c1bdd..85fbfd2a 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -330,6 +330,8 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-registration.service', 'priority': 4000, 'groups': ['matrix', 'registration']}] if matrix_registration_enabled else []) + + ([{'name': 'matrix-sliding-sync.service', 'priority': 4000, 'groups': ['matrix', 'sliding-sync']}] if matrix_sliding_sync_enabled else []) + + ([{'name': 'matrix-sygnal.service', 'priority': 800, 'groups': ['matrix', 'sygnal']}] if matrix_sygnal_enabled else []) + ([{'name': 'matrix-goofys.service', 'priority': 800, 'groups': ['matrix', 'goofys']}] if matrix_s3_media_store_enabled else []) @@ -2612,6 +2614,8 @@ matrix_nginx_proxy_systemd_wanted_services_list: | + (['matrix-jitsi.service'] if matrix_jitsi_enabled else []) + + (['matrix-sliding-sync-proxy.service'] if matrix_sliding_sync_enabled else []) + + (['matrix-bot-go-neb.service'] if matrix_bot_go_neb_enabled else []) + ([etherpad_identifier + '.service'] if etherpad_enabled else []) @@ -2745,6 +2749,12 @@ devture_postgres_managed_databases_auto: | 'password': matrix_dendrite_database_password, }] if (matrix_dendrite_enabled and matrix_dendrite_database_hostname == devture_postgres_connection_hostname) else []) + + ([{ + 'name': matrix_sliding_sync_database_name, + 'username': matrix_sliding_sync_database_username, + 'password': matrix_sliding_sync_database_password, + }] if (matrix_sliding_sync_enabled) else []) + + ([{ 'name': matrix_ma1sd_database_name, 'username': matrix_ma1sd_database_username, @@ -3639,6 +3649,63 @@ matrix_registration_database_password: "{{ '%s' | format(matrix_homeserver_gener ###################################################################### + +###################################################################### +# +# matrix-sliding-sync +# +###################################################################### + +# We don't enable the sliding sync proxy by default. +matrix_sliding_sync_enabled: false + +matrix_sliding_sync_scheme: "{{ 'https' if matrix_playbook_ssl_enabled else 'http' }}" + +matrix_sliding_sync_hostname: "{{ matrix_server_fqn_matrix }}" + +matrix_sliding_sync_path_prefix: /sliding-sync + +matrix_sliding_sync_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" + +matrix_sliding_sync_container_additional_networks: | + {{ + ( + ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + + + ([matrix_nginx_proxy_container_network] if matrix_nginx_proxy_enabled and matrix_nginx_proxy_container_network != matrix_sliding_sync_container_network else []) + + + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != matrix_sliding_sync_container_network else []) + ) | unique + }} + +matrix_sliding_sync_container_labels_traefik_enabled: "{{ matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] }}" +matrix_sliding_sync_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_sliding_sync_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_sliding_sync_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" + +matrix_sliding_sync_systemd_required_services_list: | + {{ + ['docker.service'] + + + ['matrix-' + matrix_homeserver_implementation + '.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + + + (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) + }} + +matrix_sliding_sync_environment_variable_syncv3_secret: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'ss.secret', rounds=655555) | to_uuid }}" + +matrix_sliding_sync_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" +matrix_sliding_sync_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'ss.db', rounds=655555) | to_uuid }}" + +###################################################################### +# +# /matrix-sliding-sync +# +###################################################################### + + ###################################################################### # # matrix-dendrite diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index e72269ee..a2b7f2b8 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -105,6 +105,7 @@ - custom/matrix-dimension - galaxy/etherpad - custom/etherpad-proxy-connect + - custom/matrix-sliding-sync - custom/matrix-email2matrix - custom/matrix-sygnal - galaxy/ntfy diff --git a/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 b/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 index 45fc9b66..45c3ea2a 100644 --- a/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 +++ b/roles/custom/matrix-base/templates/static-files/well-known/matrix-client.j2 @@ -26,6 +26,11 @@ "preferredDomain": {{ matrix_client_element_jitsi_preferredDomain|to_json }} } {% endif %} + {% if matrix_sliding_sync_enabled %}, + "org.matrix.msc3575.proxy": { + "url": "{{ matrix_sliding_sync_base_url }}" + } + {% endif %} {% if matrix_client_element_location_sharing_enabled %}, "m.tile_server": { "map_style_url": "https://{{ matrix_server_fqn_element }}/map_style.json" diff --git a/roles/custom/matrix-sliding-sync/defaults/main.yml b/roles/custom/matrix-sliding-sync/defaults/main.yml new file mode 100644 index 00000000..f66b3b61 --- /dev/null +++ b/roles/custom/matrix-sliding-sync/defaults/main.yml @@ -0,0 +1,91 @@ +--- + +# Sliding Sync Proxy is an implementation of MSC3575 for the new sliding sync + +matrix_sliding_sync_enabled: true + +matrix_sliding_sync_version: v0.99.1 + +matrix_sliding_sync_scheme: https + +# The hostname at which sliding-sync is served. +matrix_sliding_sync_hostname: '' + +# The path at which sliding-sync is served. +# This value must either be `/` or not end with a slash (e.g. `/sliding-sync`). +matrix_sliding_sync_path_prefix: / + +matrix_sliding_sync_base_path: "{{ matrix_base_data_path }}/sliding-sync" + +matrix_sliding_sync_container_image_self_build: false +matrix_sliding_sync_container_image_self_build_repo: https://github.com/matrix-org/sliding-sync +matrix_sliding_sync_container_image_self_build_repo_version: "{{ 'main' if matrix_sliding_sync_version == 'main' else matrix_sliding_sync_version }}" +matrix_sliding_sync_container_src_path: "{{ matrix_sliding_sync_base_path }}/container-src" + +matrix_sliding_sync_container_image: "{{ matrix_sliding_sync_container_image_name_prefix }}matrix-org/sliding-sync:{{ matrix_sliding_sync_container_image_tag }}" +matrix_sliding_sync_container_image_name_prefix: "{{ 'localhost/' if matrix_sliding_sync_container_image_self_build else matrix_sliding_sync_container_image_registry_prefix }}" +matrix_sliding_sync_container_image_tag: "{{ matrix_sliding_sync_version }}" +matrix_sliding_sync_container_image_force_pull: "{{ matrix_sliding_sync_container_image.endswith(':main') }}" +matrix_sliding_sync_container_image_registry_prefix: ghcr.io/ + +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_sliding_sync_container_network: matrix-sliding-sync + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_sliding_sync_container_additional_networks: [] + +# matrix_sliding_sync_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_sliding_sync_container_labels_additional_labels`. +matrix_sliding_sync_container_labels_traefik_enabled: true +matrix_sliding_sync_container_labels_traefik_docker_network: "{{ matrix_sliding_sync_container_network }}" +matrix_sliding_sync_container_labels_traefik_hostname: "{{ matrix_sliding_sync_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/sliding-sync`). +matrix_sliding_sync_container_labels_traefik_path_prefix: "{{ matrix_sliding_sync_path_prefix }}" +matrix_sliding_sync_container_labels_traefik_rule: "Host(`{{ matrix_sliding_sync_container_labels_traefik_hostname }}`){% if matrix_sliding_sync_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_sliding_sync_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_sliding_sync_container_labels_traefik_priority: 0 +matrix_sliding_sync_container_labels_traefik_entrypoints: web-secure +matrix_sliding_sync_container_labels_traefik_tls: "{{ matrix_sliding_sync_container_labels_traefik_entrypoints != 'web' }}" +matrix_sliding_sync_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_sliding_sync_container_labels_traefik_additional_response_headers_custom` +matrix_sliding_sync_container_labels_traefik_additional_response_headers: "{{ matrix_sliding_sync_container_labels_traefik_additional_response_headers_auto | combine(matrix_sliding_sync_container_labels_traefik_additional_response_headers_custom) }}" +matrix_sliding_sync_container_labels_traefik_additional_response_headers_auto: {} +matrix_sliding_sync_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_sliding_sync_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_sliding_sync_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_sliding_sync_container_labels_additional_labels: '' + +# A list of extra arguments to pass to the container +matrix_sliding_sync_container_extra_arguments: [] + +# List of systemd services that matrix-sliding-sync-proxy.service depends on +matrix_sliding_sync_systemd_required_services_list: ["docker.service"] + +# List of systemd services that matrix-sliding-sync-proxy.service wants +matrix_sliding_sync_systemd_wanted_services_list: [] + +# Controls the SYNCV3_SERVER environment variable +matrix_sliding_sync_environment_variable_syncv3_server: "{{ matrix_homeserver_container_url }}" + +# Controls the SYNCV3_SECRET environment variable +matrix_sliding_sync_environment_variable_syncv3_secret: '' + +# Controls the SYNCV3_DB environment variable +matrix_sliding_sync_environment_variable_syncv3_db: 'user={{ matrix_sliding_sync_database_username }} password={{ matrix_sliding_sync_database_password }} host={{ matrix_sliding_sync_database_hostname }} port={{ matrix_sliding_sync_database_port }} dbname={{ matrix_sliding_sync_database_name }} sslmode=disable' + +matrix_sliding_sync_database_username: 'matrix_sliding_sync' +matrix_sliding_sync_database_password: '' +matrix_sliding_sync_database_hostname: '' +matrix_sliding_sync_database_port: 5432 +matrix_sliding_sync_database_name: 'matrix_sliding_sync' diff --git a/roles/custom/matrix-sliding-sync/tasks/install.yml b/roles/custom/matrix-sliding-sync/tasks/install.yml new file mode 100644 index 00000000..26717afd --- /dev/null +++ b/roles/custom/matrix-sliding-sync/tasks/install.yml @@ -0,0 +1,69 @@ +--- + +- name: Ensure matrix-sliding-sync paths exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - path: "{{ matrix_sliding_sync_base_path }}" + when: true + - path: "{{ matrix_sliding_sync_container_src_path }}" + when: "{{ matrix_sliding_sync_container_image_self_build }}" + when: item.when | bool + +- name: Ensure matrix-sliding-sync support files installed + ansible.builtin.template: + src: "{{ role_path }}/templates/{{ item }}.j2" + dest: "{{ matrix_sliding_sync_base_path }}/{{ item }}" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - env + - labels + +- name: Ensure matrix-sliding-sync container image is pulled + community.docker.docker_image: + name: "{{ matrix_sliding_sync_container_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_sliding_sync_container_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_sliding_sync_container_image_force_pull }}" + when: "not matrix_sliding_sync_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- when: matrix_sliding_sync_container_image_self_build | bool + block: + - name: Ensure matrix-sliding-sync repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_sliding_sync_container_image_self_build_repo }}" + version: "{{ matrix_sliding_sync_container_image_self_build_repo_version }}" + dest: "{{ matrix_sliding_sync_container_src_path }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_sliding_sync_git_pull_results + + - name: Ensure matrix-sliding-sync container image is built + ansible.builtin.command: + cmd: |- + {{ devture_systemd_docker_base_host_command_docker }} buildx build + --tag={{ matrix_sliding_sync_container_image }} + --file={{ matrix_sliding_sync_container_src_path }}/Dockerfile + {{ matrix_sliding_sync_container_src_path }} + +- name: Ensure matrix-sliding-sync container network is created + community.general.docker_network: + name: "{{ matrix_sliding_sync_container_network }}" + driver: bridge + +- name: Ensure matrix-sliding-sync.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-sliding-sync.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-sliding-sync.service" + mode: 0644 diff --git a/roles/custom/matrix-sliding-sync/tasks/main.yml b/roles/custom/matrix-sliding-sync/tasks/main.yml new file mode 100644 index 00000000..71e7391b --- /dev/null +++ b/roles/custom/matrix-sliding-sync/tasks/main.yml @@ -0,0 +1,19 @@ +--- +- block: + - when: matrix_sliding_sync_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_sliding_sync_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" + tags: + - setup-all + - setup-sliding-sync + - install-all + - install-sliding-sync + +- block: + - when: not matrix_sliding_sync_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" + tags: + - setup-all + - setup-sliding-sync diff --git a/roles/custom/matrix-sliding-sync/tasks/uninstall.yml b/roles/custom/matrix-sliding-sync/tasks/uninstall.yml new file mode 100644 index 00000000..8a559cac --- /dev/null +++ b/roles/custom/matrix-sliding-sync/tasks/uninstall.yml @@ -0,0 +1,25 @@ +--- + +- name: Check existence of matrix-sliding-sync service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-sliding-sync.service" + register: matrix_sliding_sync_service_stat + +- when: matrix_sliding_sync_service_stat.stat.exists | bool + block: + - name: Ensure matrix-sliding-sync is stopped + ansible.builtin.service: + name: matrix-sliding-sync + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-sliding-sync.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-sliding-sync.service" + state: absent + + - name: Ensure matrix-sliding-sync paths don't exist + ansible.builtin.file: + path: "{{ matrix_sliding_sync_base_path }}" + state: absent diff --git a/roles/custom/matrix-sliding-sync/tasks/validate_config.yml b/roles/custom/matrix-sliding-sync/tasks/validate_config.yml new file mode 100644 index 00000000..5d36403c --- /dev/null +++ b/roles/custom/matrix-sliding-sync/tasks/validate_config.yml @@ -0,0 +1,11 @@ +--- +- name: Fail if required matrix-sliding-sync settings not defined + ansible.builtin.fail: + msg: > + You need to define a required configuration setting (`{{ item.name }}`). + when: "vars[item] == ''" + with_items: + - matrix_sliding_sync_hostname + - matrix_sliding_sync_path_prefix + - matrix_sliding_sync_database_hostname + - matrix_sliding_sync_environment_variable_syncv3_secret diff --git a/roles/custom/matrix-sliding-sync/templates/env.j2 b/roles/custom/matrix-sliding-sync/templates/env.j2 new file mode 100644 index 00000000..1269bd2a --- /dev/null +++ b/roles/custom/matrix-sliding-sync/templates/env.j2 @@ -0,0 +1,4 @@ +SYNCV3_SERVER={{ matrix_sliding_sync_environment_variable_syncv3_server }} +SYNCV3_SECRET={{ matrix_sliding_sync_environment_variable_syncv3_secret }} +SYNCV3_BINDADDR=:8008 +SYNCV3_DB={{ matrix_sliding_sync_environment_variable_syncv3_db }} diff --git a/roles/custom/matrix-sliding-sync/templates/labels.j2 b/roles/custom/matrix-sliding-sync/templates/labels.j2 new file mode 100644 index 00000000..66549249 --- /dev/null +++ b/roles/custom/matrix-sliding-sync/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_sliding_sync_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_sliding_sync_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_sliding_sync_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_sliding_sync_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-sliding-sync-slashless-redirect.redirectregex.regex=({{ matrix_sliding_sync_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-sliding-sync-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-sliding-sync-slashless-redirect'] %} +{% endif %} + +{% if matrix_sliding_sync_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-sliding-sync-strip-prefix.stripprefix.prefixes={{ matrix_sliding_sync_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-sliding-sync-strip-prefix'] %} +{% endif %} + +{% if matrix_sliding_sync_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_sliding_sync_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-sliding-sync-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-sliding-sync-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-sliding-sync.rule={{ matrix_sliding_sync_container_labels_traefik_rule }} +{% if matrix_sliding_sync_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-sliding-sync.priority={{ matrix_sliding_sync_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-sliding-sync.service=matrix-sliding-sync +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-sliding-sync.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-sliding-sync.entrypoints={{ matrix_sliding_sync_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-sliding-sync.tls={{ matrix_sliding_sync_container_labels_traefik_tls | to_json }} +{% if matrix_sliding_sync_container_labels_traefik_tls %} +traefik.http.routers.matrix-sliding-sync.tls.certResolver={{ matrix_sliding_sync_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-sliding-sync.loadbalancer.server.port=8008 +{% endif %} + +{{ matrix_sliding_sync_container_labels_additional_labels }} diff --git a/roles/custom/matrix-sliding-sync/templates/systemd/matrix-sliding-sync.service.j2 b/roles/custom/matrix-sliding-sync/templates/systemd/matrix-sliding-sync.service.j2 new file mode 100644 index 00000000..ac8d2ffb --- /dev/null +++ b/roles/custom/matrix-sliding-sync/templates/systemd/matrix-sliding-sync.service.j2 @@ -0,0 +1,49 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=matrix-sliding-sync +{% for service in matrix_sliding_sync_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_sliding_sync_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-sliding-sync 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-sliding-sync 2>/dev/null || true' + +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-sliding-sync \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --tmpfs /tmp \ + --network={{ matrix_sliding_sync_container_network }} \ + --env-file={{ matrix_sliding_sync_base_path }}/env \ + --label-file={{ matrix_sliding_sync_base_path }}/labels \ + {% for arg in matrix_sliding_sync_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_sliding_sync_container_image }} + +{% for network in matrix_sliding_sync_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-sliding-sync +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-sliding-sync + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-sliding-sync 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-sliding-sync 2>/dev/null || true' + +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-sliding-sync + +[Install] +WantedBy=multi-user.target diff --git a/roles/custom/matrix-sliding-sync/vars/main.yml b/roles/custom/matrix-sliding-sync/vars/main.yml new file mode 100644 index 00000000..8ec69c08 --- /dev/null +++ b/roles/custom/matrix-sliding-sync/vars/main.yml @@ -0,0 +1,5 @@ +--- + +# Public facing base URL of the Sliding Sync service. +# It should not end with a slash. +matrix_sliding_sync_base_url: "{{ matrix_sliding_sync_scheme }}://{{ matrix_sliding_sync_hostname }}{{ '' if matrix_sliding_sync_path_prefix == '/' else matrix_sliding_sync_path_prefix }}" From 6c1a39e6e9efbe0c1637006c8be16238e70845b3 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 7 Mar 2023 12:22:04 +0200 Subject: [PATCH 304/382] Announce sliding-sync support Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2515 --- CHANGELOG.md | 11 +++++ README.md | 1 + ...configuring-playbook-sliding-sync-proxy.md | 47 +++++++++++++++++++ docs/configuring-playbook.md | 2 + 4 files changed, 61 insertions(+) create mode 100644 docs/configuring-playbook-sliding-sync-proxy.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 1aedacc3..49262d63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,14 @@ +# 2023-03-07 + +## Sliding Sync Proxy (Element X) support + +Thanks to [Benjamin Kampmann](https://github.com/gnunicorn) for [getting it started](https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2515), [FSG-Cat](https://github.com/FSG-Cat) for fixing it up and me ([Slavi](https://github.com/spantaleev)) for polishing it up, the playbook can now install and configure the [sliding-sync proxy](https://github.com/matrix-org/sliding-sync). + +The new Element X clients ([Element X iOS](https://github.com/vector-im/element-x-ios) and [Element X Android](https://github.com/vector-im/element-x-android)) require the `sliding-sync` proxy to do their job. These clients are still in beta, and now playbook users can easily give them a try and help test them thanks to us having `sliding-sync` support. + +To get started, see our [Setting up Sliding Sync Proxy](docs/configuring-playbook-sliding-sync-proxy.md) documentation page. + + # 2023-03-02 ## The matrix-etherpad role lives independently now diff --git a/README.md b/README.md index bd37d23e..185c4085 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,7 @@ Various services that don't fit any other category. | Name | Default? | Description | Documentation | | ---- | -------- | ----------- | ------------- | +| [sliding-sync](https://github.com/matrix-org/sliding-sync)| x | Sliding Sync support for clients which require it (e.g. Element X) | [Link](docs/configuring-playbook-sliding-sync-proxy.md) | | [synapse-simple-antispam](https://github.com/t2bot/synapse-simple-antispam) (advanced) | x | A spam checker module | [Link](docs/configuring-playbook-synapse-simple-antispam.md) | | [Matrix Corporal](https://github.com/devture/matrix-corporal) (advanced) | x | Reconciliator and gateway for a managed Matrix server | [Link](docs/configuring-playbook-matrix-corporal.md) | | [Etherpad](https://etherpad.org) | x | An open source collaborative text editor | [Link](docs/configuring-playbook-etherpad.md) | diff --git a/docs/configuring-playbook-sliding-sync-proxy.md b/docs/configuring-playbook-sliding-sync-proxy.md new file mode 100644 index 00000000..c07d1db4 --- /dev/null +++ b/docs/configuring-playbook-sliding-sync-proxy.md @@ -0,0 +1,47 @@ +# Setting up Sliding Sync Proxy (optional) + +The playbook can install and configure [sliding-sync](https://github.com/matrix-org/sliding-sync) proxy for you. + +Sliding Sync is an implementation of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md) and a prerequisite for running the new (still beta) Element X clients ([Element X iOS](https://github.com/vector-im/element-x-ios) and [Element X Android](https://github.com/vector-im/element-x-android)). + +See the project's [documentation](https://github.com/matrix-org/sliding-syncb) to learn more. + +**NOTE**: The Sliding Sync proxy **only works with the Traefik reverse-proxy**. If you have an old server installation (from the time `matrix-nginx-proxy` was our default reverse-proxy - `matrix_playbook_reverse_proxy_type: playbook-managed-nginx`), you won't be able to use Sliding Sync. + + +## Decide on a domain and path + +By default, the Sliding Sync proxy is configured to be served on the Matrix domain (`matrix.DOMAIN`, controlled by `matrix_server_fqn_matrix`), under the `/sliding-sync` path. + +This makes it easy to set it up, **without** having to [adjust your DNS records](#adjusting-dns-records). + +If you'd like to run the Sliding Sync proxy on another hostname or path, use the `matrix_sliding_sync_hostname` and `matrix_sliding_sync_path_prefix` variables. + + +## Adjusting DNS records + +If you've changed the default hostame, **you may need to adjust your DNS** records. + + +## Adjusting the playbook configuration + +Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file: + +```yaml +matrix_sliding_sync_enabled: true +``` + + +## Installing + +After potentially [adjusting DNS records](#adjusting-dns-records) and configuring the playbook, run the [installation](installing.md) command again: `just install-all`. + + +## Usage + +You **don't need to do anything special** to make use of the Sliding Sync Proxy. +Simply open your client which supports Sliding Sync (like Element X) and log in. + +When the Sliding Sync proxy is [installed](#installing), your `/.well-known/matrix/client` file is also updated. A new `org.matrix.msc3575.proxy` section and `url` property are added there and made to point to your Sliding Sync proxy's base URL (e.g. `https://matrix.DOMAIN/sliding-sync`). + +This allows clients which support Sliding Sync to detect the Sliding Sync Proxy's URL and make use of it. diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index b4bae786..68a2658a 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -195,6 +195,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins ### Other specialized services +- [Setting up the Sliding Sync Proxy](configuring-playbook-sliding-sync-proxy.md) for clients which require Sliding Sync support (like Element X) (optional) + - [Setting up the Sygnal push gateway](configuring-playbook-sygnal.md) (optional) - [Setting up the ntfy push notifications server](configuring-playbook-ntfy.md) (optional) From bf95204860e79e3e7821fd2380d63e461a74076a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 7 Mar 2023 14:44:30 +0200 Subject: [PATCH 305/382] Mention Element X Android not being quite ready for testing yet --- CHANGELOG.md | 2 +- docs/configuring-playbook-sliding-sync-proxy.md | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 49262d63..7e9e2545 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ Thanks to [Benjamin Kampmann](https://github.com/gnunicorn) for [getting it started](https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2515), [FSG-Cat](https://github.com/FSG-Cat) for fixing it up and me ([Slavi](https://github.com/spantaleev)) for polishing it up, the playbook can now install and configure the [sliding-sync proxy](https://github.com/matrix-org/sliding-sync). -The new Element X clients ([Element X iOS](https://github.com/vector-im/element-x-ios) and [Element X Android](https://github.com/vector-im/element-x-android)) require the `sliding-sync` proxy to do their job. These clients are still in beta, and now playbook users can easily give them a try and help test them thanks to us having `sliding-sync` support. +The upcoming Element X clients ([Element X iOS](https://github.com/vector-im/element-x-ios) and [Element X Android](https://github.com/vector-im/element-x-android)) require the `sliding-sync` proxy to do their job. **These clients are still in beta** (especially Element X Android, which requires manual compilation to get it working with a non-`matrix.org` homeseserver). Playbook users can now easily give these clients a try and help test them thanks to us having `sliding-sync` support. To get started, see our [Setting up Sliding Sync Proxy](docs/configuring-playbook-sliding-sync-proxy.md) documentation page. diff --git a/docs/configuring-playbook-sliding-sync-proxy.md b/docs/configuring-playbook-sliding-sync-proxy.md index c07d1db4..49e0483a 100644 --- a/docs/configuring-playbook-sliding-sync-proxy.md +++ b/docs/configuring-playbook-sliding-sync-proxy.md @@ -2,9 +2,13 @@ The playbook can install and configure [sliding-sync](https://github.com/matrix-org/sliding-sync) proxy for you. -Sliding Sync is an implementation of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md) and a prerequisite for running the new (still beta) Element X clients ([Element X iOS](https://github.com/vector-im/element-x-ios) and [Element X Android](https://github.com/vector-im/element-x-android)). +Sliding Sync is an implementation of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md) and a prerequisite for running the new (**still beta**) Element X clients ([Element X iOS](https://github.com/vector-im/element-x-ios) and [Element X Android](https://github.com/vector-im/element-x-android)). -See the project's [documentation](https://github.com/matrix-org/sliding-syncb) to learn more. +See the project's [documentation](https://github.com/matrix-org/sliding-sync) to learn more. + +Element X iOS is [available on TestFlight](https://testflight.apple.com/join/uZbeZCOi). + +Element X Android requires manual compilation to get it working with a non-`matrix.org` homeseserver. It's also less feature-complete than the iOS version. **NOTE**: The Sliding Sync proxy **only works with the Traefik reverse-proxy**. If you have an old server installation (from the time `matrix-nginx-proxy` was our default reverse-proxy - `matrix_playbook_reverse_proxy_type: playbook-managed-nginx`), you won't be able to use Sliding Sync. From dddfee16bc852fc989b057bd1867f0e85825846d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 7 Mar 2023 17:28:10 +0200 Subject: [PATCH 306/382] Fix all 300+ ansible-lint-reported errors --- .../etherpad-proxy-connect/tasks/main.yml | 8 +-- .../tasks/validate_config.yml | 16 +++--- roles/custom/matrix-aux/tasks/main.yml | 6 +- roles/custom/matrix-base/tasks/main.yml | 24 ++++---- .../matrix-bot-buscarron/tasks/main.yml | 18 +++--- .../custom/matrix-bot-chatgpt/tasks/main.yml | 19 ++++--- .../custom/matrix-bot-draupnir/tasks/main.yml | 18 +++--- roles/custom/matrix-bot-go-neb/tasks/main.yml | 18 +++--- .../custom/matrix-bot-honoroit/tasks/main.yml | 18 +++--- .../tasks/main.yml | 18 +++--- .../tasks/main.yml | 18 +++--- roles/custom/matrix-bot-maubot/tasks/main.yml | 26 ++++----- .../custom/matrix-bot-mjolnir/tasks/main.yml | 18 +++--- .../matrix-bot-postmoogle/tasks/main.yml | 18 +++--- .../tasks/main.yml | 18 +++--- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 18 +++--- .../tasks/setup_install.yml | 10 +--- .../tasks/main.yml | 18 +++--- .../tasks/main.yml | 26 ++++----- .../tasks/main.yml | 26 ++++----- .../tasks/main.yml | 18 +++--- .../tasks/main.yml | 18 +++--- .../tasks/setup_install.yml | 10 +++- .../matrix-bridge-heisenbridge/tasks/main.yml | 16 +++--- .../matrix-bridge-hookshot/defaults/main.yml | 10 ++-- .../matrix-bridge-hookshot/tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 4 +- .../tasks/main.yml | 18 +++--- .../tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 26 ++++----- .../tasks/main.yml | 26 ++++----- .../tasks/main.yml | 18 +++--- .../tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 26 ++++----- .../tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 10 +++- .../tasks/main.yml | 18 +++--- .../tasks/main.yml | 18 +++--- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 18 +++--- .../tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 18 +++--- .../tasks/setup_install.yml | 5 +- .../tasks/main.yml | 26 ++++----- .../tasks/setup_install.yml | 5 +- roles/custom/matrix-bridge-sms/tasks/main.yml | 18 +++--- .../matrix-cactus-comments/tasks/main.yml | 26 ++++----- .../custom/matrix-client-cinny/tasks/main.yml | 24 ++++---- .../matrix-client-element/tasks/main.yml | 24 ++++---- .../tasks/setup_install.yml | 1 + .../matrix-client-hydrogen/tasks/main.yml | 24 ++++---- .../custom/matrix-common-after/tasks/main.yml | 6 +- roles/custom/matrix-conduit/tasks/main.yml | 16 +++--- roles/custom/matrix-corporal/tasks/main.yml | 24 ++++---- roles/custom/matrix-coturn/tasks/main.yml | 18 +++--- .../matrix-coturn/tasks/setup_install.yml | 4 +- roles/custom/matrix-dendrite/tasks/main.yml | 30 +++++----- .../matrix-dendrite/tasks/setup_install.yml | 24 ++++---- roles/custom/matrix-dimension/tasks/main.yml | 18 +++--- .../custom/matrix-dynamic-dns/tasks/main.yml | 18 +++--- .../custom/matrix-email2matrix/tasks/main.yml | 18 +++--- roles/custom/matrix-jitsi/tasks/main.yml | 56 +++++++++---------- .../util/setup_jitsi_auth_uvs_install.yml | 5 ++ .../tasks/main.yml | 26 ++++----- roles/custom/matrix-ma1sd/tasks/main.yml | 24 ++++---- .../matrix-ma1sd/tasks/setup_install.yml | 1 + roles/custom/matrix-mailer/tasks/main.yml | 16 +++--- .../custom/matrix-nginx-proxy/tasks/main.yml | 10 ++-- ...tup_ssl_lets_encrypt_obtain_for_domain.yml | 2 + ...etup_ssl_self_signed_obtain_for_domain.yml | 21 ++++--- .../tasks/main.yml | 24 ++++---- .../tasks/main.yml | 18 +++--- roles/custom/matrix-prometheus/tasks/main.yml | 18 +++--- .../custom/matrix-rageshake/tasks/install.yml | 1 + roles/custom/matrix-rageshake/tasks/main.yml | 19 ++++--- .../custom/matrix-registration/tasks/main.yml | 38 ++++++------- .../matrix-sliding-sync/tasks/install.yml | 1 + .../custom/matrix-sliding-sync/tasks/main.yml | 19 ++++--- roles/custom/matrix-sygnal/tasks/main.yml | 18 +++--- .../matrix-synapse-admin/tasks/main.yml | 26 ++++----- .../tasks/main.yml | 16 +++--- .../tasks/ext/setup_install.yml | 56 +++++++++---------- .../tasks/ext/setup_uninstall.yml | 48 ++++++++-------- roles/custom/matrix-synapse/tasks/main.yml | 56 +++++++++---------- .../matrix-synapse/tasks/setup_install.yml | 26 ++++----- .../matrix-synapse/tasks/setup_uninstall.yml | 26 ++++----- .../tasks/synapse/setup_install.yml | 33 +++++------ .../custom/matrix-user-creator/tasks/main.yml | 8 +-- .../tasks/main.yml | 18 +++--- .../matrix_playbook_migration/tasks/main.yml | 16 +++--- 98 files changed, 935 insertions(+), 888 deletions(-) diff --git a/roles/custom/etherpad-proxy-connect/tasks/main.yml b/roles/custom/etherpad-proxy-connect/tasks/main.yml index 67bf7d0e..630ab87f 100644 --- a/roles/custom/etherpad-proxy-connect/tasks/main.yml +++ b/roles/custom/etherpad-proxy-connect/tasks/main.yml @@ -1,12 +1,12 @@ --- - when: etherpad_enabled | bool and etherpad_nginx_proxy_dimension_integration_enabled | bool - block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" tags: - install-all - setup-all - install-nginx-proxy - setup-nginx-proxy + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" diff --git a/roles/custom/etherpad-proxy-connect/tasks/validate_config.yml b/roles/custom/etherpad-proxy-connect/tasks/validate_config.yml index 2127ce1a..51a672ee 100644 --- a/roles/custom/etherpad-proxy-connect/tasks/validate_config.yml +++ b/roles/custom/etherpad-proxy-connect/tasks/validate_config.yml @@ -1,30 +1,30 @@ --- -- when: matrix_playbook_reverse_proxy_type not in ['playbook-managed-nginx', 'other-nginx-non-container'] - name: Fail if reverse-proxy is not nginx +- name: Fail if reverse-proxy is not nginx + when: matrix_playbook_reverse_proxy_type not in ['playbook-managed-nginx', 'other-nginx-non-container'] ansible.builtin.fail: msg: > Etherpad's integration into matrix-nginx-proxy's Dimension server only makes sense if you're using matrix-nginx-proxy. `matrix_playbook_reverse_proxy_type` ({{ matrix_playbook_reverse_proxy_type }}) indicates that you're using another reverse-proxy. If you're using Traefik, you should configure `etherpad_hostname` and `etherpad_path_prefix` instead. -- when: not matrix_dimension_enabled - name: Fail if Dimension not enabled +- name: Fail if Dimension not enabled + when: not matrix_dimension_enabled ansible.builtin.fail: msg: > Etherpad's integration into matrix-nginx-proxy's Dimension server only makes sense if you're using Dimension. Looks like Dimension is not enabled in your configuration (judging by `matrix_dimension_enabled`). Consider configuring `etherpad_hostname` and `etherpad_path_prefix` instead. -- when: etherpad_hostname != matrix_server_fqn_dimension - name: Fail if Etherpad hostname does not match Dimension hostname +- name: Fail if Etherpad hostname does not match Dimension hostname + when: etherpad_hostname != matrix_server_fqn_dimension ansible.builtin.fail: msg: > Etherpad's integration into matrix-nginx-proxy's Dimension server requires that you set `etherpad_hostname` to `matrix_server_fqn_dimension`. Consider adding this to your configuration: `{% raw %}etherpad_hostname: "{{ matrix_server_fqn_dimension }}"{% endraw %}` -- when: etherpad_nginx_proxy_dimension_integration_path_prefix == '/' - name: Fail if / path prefix used for Etherpad +- name: Fail if / path prefix used for Etherpad + when: etherpad_nginx_proxy_dimension_integration_path_prefix == '/' ansible.builtin.fail: msg: > Etherpad's integration into matrix-nginx-proxy's Dimension server only makes sense if you're using a non-`/` path for Etherpad. diff --git a/roles/custom/matrix-aux/tasks/main.yml b/roles/custom/matrix-aux/tasks/main.yml index 57db0965..3ae8d179 100644 --- a/roles/custom/matrix-aux/tasks/main.yml +++ b/roles/custom/matrix-aux/tasks/main.yml @@ -1,9 +1,9 @@ --- -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup.yml" - tags: +- tags: - setup-all - setup-aux-files - install-all - install-aux-files + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup.yml" diff --git a/roles/custom/matrix-base/tasks/main.yml b/roles/custom/matrix-base/tasks/main.yml index c67e2090..13ef11d5 100644 --- a/roles/custom/matrix-base/tasks/main.yml +++ b/roles/custom/matrix-base/tasks/main.yml @@ -1,30 +1,28 @@ --- -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - tags: +- tags: - setup-all - install-all + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" # This needs to always run, because it populates `matrix_user_uid` and `matrix_user_gid`, # which are required by many other roles. -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_matrix_user.yml" - tags: +- tags: - always - setup-system-user - common + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_matrix_user.yml" -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_matrix_base.yml" - tags: +- tags: - setup-all - install-all - common + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_matrix_base.yml" -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_well_known.yml" - tags: +- tags: - setup-all - setup-ma1sd - setup-synapse @@ -37,3 +35,5 @@ - install-dendrite - install-conduit - install-nginx-proxy + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_well_known.yml" diff --git a/roles/custom/matrix-bot-buscarron/tasks/main.yml b/roles/custom/matrix-bot-buscarron/tasks/main.yml index b4a58e91..fbf5c96a 100644 --- a/roles/custom/matrix-bot-buscarron/tasks/main.yml +++ b/roles/custom/matrix-bot-buscarron/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-buscarron + - install-all + - install-bot-buscarron + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" when: "run_setup | bool and matrix_bot_buscarron_enabled | bool" - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" when: "run_setup | bool and matrix_bot_buscarron_enabled | bool" - tags: + +- tags: - setup-all - setup-bot-buscarron - - install-all - - install-bot-buscarron - -- block: + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" when: "run_setup | bool and not matrix_bot_buscarron_enabled | bool" - tags: - - setup-all - - setup-bot-buscarron diff --git a/roles/custom/matrix-bot-chatgpt/tasks/main.yml b/roles/custom/matrix-bot-chatgpt/tasks/main.yml index c5487fb8..1086a62d 100644 --- a/roles/custom/matrix-bot-chatgpt/tasks/main.yml +++ b/roles/custom/matrix-bot-chatgpt/tasks/main.yml @@ -1,20 +1,21 @@ --- -- block: +- + tags: + - setup-all + - setup-bot-chatgpt + - install-all + - install-bot-chatgpt + block: - when: matrix_bot_chatgpt_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_chatgpt_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-bot-chatgpt - - install-all - - install-bot-chatgpt - -- block: + block: - when: not matrix_bot_chatgpt_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-bot-chatgpt diff --git a/roles/custom/matrix-bot-draupnir/tasks/main.yml b/roles/custom/matrix-bot-draupnir/tasks/main.yml index 686fe298..66c7cd06 100644 --- a/roles/custom/matrix-bot-draupnir/tasks/main.yml +++ b/roles/custom/matrix-bot-draupnir/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-draupnir + - install-all + - install-bot-draupnir + block: - when: matrix_bot_draupnir_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_draupnir_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-bot-draupnir - - install-all - - install-bot-draupnir - -- block: + block: - when: not matrix_bot_draupnir_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-bot-draupnir diff --git a/roles/custom/matrix-bot-go-neb/tasks/main.yml b/roles/custom/matrix-bot-go-neb/tasks/main.yml index 3249fc7b..9367419d 100644 --- a/roles/custom/matrix-bot-go-neb/tasks/main.yml +++ b/roles/custom/matrix-bot-go-neb/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-go-neb + - install-all + - install-bot-go-neb + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" when: "run_setup | bool and matrix_bot_go_neb_enabled | bool" - ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" when: "run_setup | bool and matrix_bot_go_neb_enabled | bool" - tags: + +- tags: - setup-all - setup-bot-go-neb - - install-all - - install-bot-go-neb - -- block: + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" when: "run_setup | bool and not matrix_bot_go_neb_enabled | bool" - tags: - - setup-all - - setup-bot-go-neb diff --git a/roles/custom/matrix-bot-honoroit/tasks/main.yml b/roles/custom/matrix-bot-honoroit/tasks/main.yml index d4dcbce4..9b4e2908 100644 --- a/roles/custom/matrix-bot-honoroit/tasks/main.yml +++ b/roles/custom/matrix-bot-honoroit/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-honoroit + - install-all + - install-bot-honoroit + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" when: "run_setup | bool and matrix_bot_honoroit_enabled | bool" - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" when: "run_setup | bool and matrix_bot_honoroit_enabled | bool" - tags: + +- tags: - setup-all - setup-bot-honoroit - - install-all - - install-bot-honoroit - -- block: + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" when: "run_setup | bool and not matrix_bot_honoroit_enabled | bool" - tags: - - setup-all - - setup-bot-honoroit diff --git a/roles/custom/matrix-bot-matrix-registration-bot/tasks/main.yml b/roles/custom/matrix-bot-matrix-registration-bot/tasks/main.yml index 90ee56cc..83291fb6 100644 --- a/roles/custom/matrix-bot-matrix-registration-bot/tasks/main.yml +++ b/roles/custom/matrix-bot-matrix-registration-bot/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-matrix-registration-bot + - install-all + - install-bot-matrix-registration-bot + block: - when: matrix_bot_matrix_registration_bot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_matrix_registration_bot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-bot-matrix-registration-bot - - install-all - - install-bot-matrix-registration-bot - -- block: + block: - when: not matrix_bot_matrix_registration_bot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-bot-matrix-registration-bot diff --git a/roles/custom/matrix-bot-matrix-reminder-bot/tasks/main.yml b/roles/custom/matrix-bot-matrix-reminder-bot/tasks/main.yml index 22c014de..f475afc0 100644 --- a/roles/custom/matrix-bot-matrix-reminder-bot/tasks/main.yml +++ b/roles/custom/matrix-bot-matrix-reminder-bot/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-matrix-reminder-bot + - install-all + - install-bot-matrix-reminder-bot + block: - when: matrix_bot_matrix_reminder_bot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_matrix_reminder_bot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-bot-matrix-reminder-bot - - install-all - - install-bot-matrix-reminder-bot - -- block: + block: - when: not matrix_bot_matrix_reminder_bot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-bot-matrix-reminder-bot diff --git a/roles/custom/matrix-bot-maubot/tasks/main.yml b/roles/custom/matrix-bot-maubot/tasks/main.yml index 95fad19e..3241795b 100644 --- a/roles/custom/matrix-bot-maubot/tasks/main.yml +++ b/roles/custom/matrix-bot-maubot/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_bot_maubot_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_bot_maubot_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-bot-maubot + - install-all + - install-bot-maubot + block: - when: matrix_bot_maubot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_maubot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-bot-maubot - - install-all - - install-bot-maubot - -- block: + block: - when: not matrix_bot_maubot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-bot-maubot diff --git a/roles/custom/matrix-bot-mjolnir/tasks/main.yml b/roles/custom/matrix-bot-mjolnir/tasks/main.yml index def5c9f0..03f5ba90 100644 --- a/roles/custom/matrix-bot-mjolnir/tasks/main.yml +++ b/roles/custom/matrix-bot-mjolnir/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-mjolnir + - install-all + - install-bot-mjolnir + block: - when: matrix_bot_mjolnir_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_mjolnir_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-bot-mjolnir - - install-all - - install-bot-mjolnir - -- block: + block: - when: not matrix_bot_mjolnir_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-bot-mjolnir diff --git a/roles/custom/matrix-bot-postmoogle/tasks/main.yml b/roles/custom/matrix-bot-postmoogle/tasks/main.yml index a6bc76a3..b7704039 100644 --- a/roles/custom/matrix-bot-postmoogle/tasks/main.yml +++ b/roles/custom/matrix-bot-postmoogle/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-bot-postmoogle + - install-all + - install-bot-postmoogle + block: - when: matrix_bot_postmoogle_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_postmoogle_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-bot-postmoogle - - install-all - - install-bot-postmoogle - -- block: + block: - when: not matrix_bot_postmoogle_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-bot-postmoogle diff --git a/roles/custom/matrix-bridge-appservice-discord/tasks/main.yml b/roles/custom/matrix-bridge-appservice-discord/tasks/main.yml index 926fe367..88a77220 100644 --- a/roles/custom/matrix-bridge-appservice-discord/tasks/main.yml +++ b/roles/custom/matrix-bridge-appservice-discord/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-appservice-discord + - install-all + - install-appservice-discord + block: - when: matrix_appservice_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_appservice_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-appservice-discord - - install-all - - install-appservice-discord - -- block: + block: - when: not matrix_appservice_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-appservice-discord diff --git a/roles/custom/matrix-bridge-appservice-discord/tasks/setup_install.yml b/roles/custom/matrix-bridge-appservice-discord/tasks/setup_install.yml index 4a6419ed..632948bc 100644 --- a/roles/custom/matrix-bridge-appservice-discord/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-appservice-discord/tasks/setup_install.yml @@ -65,7 +65,10 @@ when: "matrix_appservice_discord_stat_db.stat.exists" - name: (Data relocation) Move AppService Discord discord.db file to ./data directory - ansible.builtin.command: "mv {{ matrix_appservice_discord_base_path }}/{{ item }} {{ matrix_appservice_discord_data_path }}/{{ item }}" + ansible.builtin.command: + cmd: "mv {{ matrix_appservice_discord_base_path }}/{{ item }} {{ matrix_appservice_discord_data_path }}/{{ item }}" + creates: "{{ matrix_appservice_discord_data_path }}/{{ item }}" + removes: "{{ matrix_appservice_discord_base_path }}/{{ item }}" with_items: - discord.db - user-store.db diff --git a/roles/custom/matrix-bridge-appservice-irc/tasks/main.yml b/roles/custom/matrix-bridge-appservice-irc/tasks/main.yml index 1cc6b35c..d4669848 100644 --- a/roles/custom/matrix-bridge-appservice-irc/tasks/main.yml +++ b/roles/custom/matrix-bridge-appservice-irc/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-appservice-irc + - install-all + - install-appservice-irc + block: - when: matrix_appservice_irc_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_appservice_irc_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-appservice-irc - - install-all - - install-appservice-irc - -- block: + block: - when: not matrix_appservice_irc_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-appservice-irc diff --git a/roles/custom/matrix-bridge-appservice-irc/tasks/setup_install.yml b/roles/custom/matrix-bridge-appservice-irc/tasks/setup_install.yml index a5fdacde..69a9aa4c 100644 --- a/roles/custom/matrix-bridge-appservice-irc/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-appservice-irc/tasks/setup_install.yml @@ -108,16 +108,12 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" -- name: Check if Appservice IRC passkey exists - ansible.builtin.stat: - path: "{{ matrix_appservice_irc_data_path }}/passkey.pem" - register: irc_passkey_file - - name: Generate Appservice IRC passkey if it doesn't exist - ansible.builtin.shell: "{{ matrix_host_command_openssl }} genpkey -out {{ matrix_appservice_irc_data_path }}/passkey.pem -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:2048" + ansible.builtin.shell: + cmd: "{{ matrix_host_command_openssl }} genpkey -out {{ matrix_appservice_irc_data_path }}/passkey.pem -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:2048" + creates: "{{ matrix_appservice_irc_data_path }}/passkey.pem" become: true become_user: "{{ matrix_user_username }}" - when: "not irc_passkey_file.stat.exists" # In the past, we used to generate the passkey.pem file with root, so permissions may not be okay. # Fix it. diff --git a/roles/custom/matrix-bridge-appservice-kakaotalk/tasks/main.yml b/roles/custom/matrix-bridge-appservice-kakaotalk/tasks/main.yml index 14a30e65..7a360024 100644 --- a/roles/custom/matrix-bridge-appservice-kakaotalk/tasks/main.yml +++ b/roles/custom/matrix-bridge-appservice-kakaotalk/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-appservice-kakaotalk + - install-all + - install-appservice-kakaotalk + block: - when: matrix_appservice_kakaotalk_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_appservice_kakaotalk_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-appservice-kakaotalk - - install-all - - install-appservice-kakaotalk - -- block: + block: - when: not matrix_appservice_kakaotalk_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-appservice-kakaotalk diff --git a/roles/custom/matrix-bridge-appservice-slack/tasks/main.yml b/roles/custom/matrix-bridge-appservice-slack/tasks/main.yml index 41e2679e..6fb8ac7c 100644 --- a/roles/custom/matrix-bridge-appservice-slack/tasks/main.yml +++ b/roles/custom/matrix-bridge-appservice-slack/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_appservice_slack_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_appservice_slack_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-appservice-slack + - install-all + - install-appservice-slack + block: - when: matrix_appservice_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_appservice_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-appservice-slack - - install-all - - install-appservice-slack - -- block: + block: - when: not matrix_appservice_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-appservice-slack diff --git a/roles/custom/matrix-bridge-appservice-webhooks/tasks/main.yml b/roles/custom/matrix-bridge-appservice-webhooks/tasks/main.yml index ae36cf70..a40d66dd 100644 --- a/roles/custom/matrix-bridge-appservice-webhooks/tasks/main.yml +++ b/roles/custom/matrix-bridge-appservice-webhooks/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_appservice_webhooks_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_appservice_webhooks_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-appservice-webhooks + - install-all + - install-appservice-webhooks + block: - when: matrix_appservice_webhooks_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_appservice_webhooks_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-appservice-webhooks - - install-all - - install-appservice-webhooks - -- block: + block: - when: not matrix_appservice_webhooks_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-appservice-webhooks diff --git a/roles/custom/matrix-bridge-beeper-linkedin/tasks/main.yml b/roles/custom/matrix-bridge-beeper-linkedin/tasks/main.yml index 50228042..56eb91e5 100644 --- a/roles/custom/matrix-bridge-beeper-linkedin/tasks/main.yml +++ b/roles/custom/matrix-bridge-beeper-linkedin/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-beeper-linkedin + - install-all + - install-beeper-linkedin + block: - when: matrix_beeper_linkedin_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_beeper_linkedin_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-beeper-linkedin - - install-all - - install-beeper-linkedin - -- block: + block: - when: not matrix_beeper_linkedin_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-beeper-linkedin diff --git a/roles/custom/matrix-bridge-go-skype-bridge/tasks/main.yml b/roles/custom/matrix-bridge-go-skype-bridge/tasks/main.yml index 5bf9236b..7ba91421 100644 --- a/roles/custom/matrix-bridge-go-skype-bridge/tasks/main.yml +++ b/roles/custom/matrix-bridge-go-skype-bridge/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-go-skype-bridge + - install-all + - install-go-skype-bridge + block: - when: matrix_go_skype_bridge_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_go_skype_bridge_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-go-skype-bridge - - install-all - - install-go-skype-bridge - -- block: + block: - when: not matrix_go_skype_bridge_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-go-skype-bridge diff --git a/roles/custom/matrix-bridge-go-skype-bridge/tasks/setup_install.yml b/roles/custom/matrix-bridge-go-skype-bridge/tasks/setup_install.yml index 6146637f..7d8ccd8d 100644 --- a/roles/custom/matrix-bridge-go-skype-bridge/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-go-skype-bridge/tasks/setup_install.yml @@ -97,11 +97,17 @@ when: "matrix_go_skype_bridge_stat_database.stat.exists" - name: (Data relocation) Move go-skype-bridge database file to ./data directory - ansible.builtin.command: "mv {{ matrix_go_skype_bridge_base_path }}/go-skype-bridge.db {{ matrix_go_skype_bridge_data_path }}/go-skype-bridge.db" + ansible.builtin.command: + cmd: "mv {{ matrix_go_skype_bridge_base_path }}/go-skype-bridge.db {{ matrix_go_skype_bridge_data_path }}/go-skype-bridge.db" + creates: "{{ matrix_go_skype_bridge_data_path }}/go-skype-bridge.db" + removes: "{{ matrix_go_skype_bridge_base_path }}/go-skype-bridge.db" when: "matrix_go_skype_bridge_stat_database.stat.exists" - name: (Data relocation) Move go-skype-bridge mx-state file to ./data directory - ansible.builtin.command: "mv {{ matrix_go_skype_bridge_base_path }}/mx-state.json {{ matrix_go_skype_bridge_data_path }}/mx-state.json" + ansible.builtin.command: + cmd: "mv {{ matrix_go_skype_bridge_base_path }}/mx-state.json {{ matrix_go_skype_bridge_data_path }}/mx-state.json" + creates: "{{ matrix_go_skype_bridge_data_path }}/mx-state.json" + removes: "{{ matrix_go_skype_bridge_base_path }}/mx-state.json" when: "matrix_go_skype_bridge_stat_mx_state.stat.exists" - name: Ensure go-skype-bridge config.yaml installed diff --git a/roles/custom/matrix-bridge-heisenbridge/tasks/main.yml b/roles/custom/matrix-bridge-heisenbridge/tasks/main.yml index 0d8354cf..5b92520d 100644 --- a/roles/custom/matrix-bridge-heisenbridge/tasks/main.yml +++ b/roles/custom/matrix-bridge-heisenbridge/tasks/main.yml @@ -1,17 +1,17 @@ --- -- block: - - when: matrix_heisenbridge_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: +- tags: - setup-all - setup-heisenbridge - install-all - install-heisenbridge + block: + - when: matrix_heisenbridge_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" -- block: - - when: not matrix_heisenbridge_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-heisenbridge + block: + - when: not matrix_heisenbridge_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" diff --git a/roles/custom/matrix-bridge-hookshot/defaults/main.yml b/roles/custom/matrix-bridge-hookshot/defaults/main.yml index aa4ae45c..3b65b92a 100644 --- a/roles/custom/matrix-bridge-hookshot/defaults/main.yml +++ b/roles/custom/matrix-bridge-hookshot/defaults/main.yml @@ -122,15 +122,15 @@ matrix_hookshot_jira_oauth_redirect_uri: "{{ matrix_hookshot_urlprefix }}{{ matr # No need to change these matrix_hookshot_generic_enabled: true -matrix_hookshot_generic_enableHttpGet: false +matrix_hookshot_generic_enableHttpGet: false # noqa var-naming # Default value of matrix_hookshot_generic_endpoint: "/hookshot/webhooks" matrix_hookshot_generic_endpoint: "{{ matrix_hookshot_webhook_endpoint }}" # urlprefix gets updated with protocol & port in group_vars/matrix_servers -matrix_hookshot_generic_urlPrefix: "{{ matrix_hookshot_urlprefix }}{{ matrix_hookshot_generic_endpoint }}" +matrix_hookshot_generic_urlPrefix: "{{ matrix_hookshot_urlprefix }}{{ matrix_hookshot_generic_endpoint }}" # noqa var-naming # If you're also using matrix-appservice-webhooks, take care that these prefixes don't overlap -matrix_hookshot_generic_userIdPrefix: '_webhooks_' -matrix_hookshot_generic_allowJsTransformationFunctions: false -matrix_hookshot_generic_waitForComplete: false +matrix_hookshot_generic_userIdPrefix: '_webhooks_' # noqa var-naming +matrix_hookshot_generic_allowJsTransformationFunctions: false # noqa var-naming +matrix_hookshot_generic_waitForComplete: false # noqa var-naming matrix_hookshot_feeds_enabled: true diff --git a/roles/custom/matrix-bridge-hookshot/tasks/main.yml b/roles/custom/matrix-bridge-hookshot/tasks/main.yml index c7ffc304..e2fa9936 100644 --- a/roles/custom/matrix-bridge-hookshot/tasks/main.yml +++ b/roles/custom/matrix-bridge-hookshot/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_hookshot_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_hookshot_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-hookshot + - install-all + - install-hookshot + block: - when: matrix_hookshot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_hookshot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-hookshot - - install-all - - install-hookshot - -- block: + block: - when: not matrix_hookshot_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-hookshot diff --git a/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml b/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml index e9b277ce..f0e8c1c6 100644 --- a/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml @@ -57,7 +57,9 @@ register: hookshot_passkey_file - name: Generate hookshot passkey if it doesn't exist - ansible.builtin.shell: "{{ matrix_host_command_openssl }} genpkey -out {{ matrix_hookshot_base_path }}/passkey.pem -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:4096" + ansible.builtin.shell: + cmd: "{{ matrix_host_command_openssl }} genpkey -out {{ matrix_hookshot_base_path }}/passkey.pem -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:4096" + creates: "{{ matrix_hookshot_base_path }}/passkey.pem" become: true become_user: "{{ matrix_user_username }}" when: "not hookshot_passkey_file.stat.exists" diff --git a/roles/custom/matrix-bridge-mautrix-discord/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-discord/tasks/main.yml index cc90f2fa..19698f9f 100644 --- a/roles/custom/matrix-bridge-mautrix-discord/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-discord/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-mautrix-discord + - install-all + - install-mautrix-discord + block: - when: matrix_mautrix_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-discord - - install-all - - install-mautrix-discord - -- block: + block: - when: not matrix_mautrix_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-discord diff --git a/roles/custom/matrix-bridge-mautrix-facebook/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-facebook/tasks/main.yml index fa7cec7d..11034668 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_facebook_enabled | bool and matrix_mautrix_facebook_appservice_public_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_facebook_enabled | bool and matrix_mautrix_facebook_appservice_public_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-facebook + - install-all + - install-mautrix-facebook + block: - when: matrix_mautrix_facebook_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_facebook_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-facebook - - install-all - - install-mautrix-facebook - -- block: + block: - when: not matrix_mautrix_facebook_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-facebook diff --git a/roles/custom/matrix-bridge-mautrix-facebook/tasks/setup_install.yml b/roles/custom/matrix-bridge-mautrix-facebook/tasks/setup_install.yml index 758a5fed..fac1d1d2 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/tasks/setup_install.yml @@ -91,7 +91,10 @@ when: "matrix_mautrix_facebook_stat_database.stat.exists" - name: (Data relocation) Move mautrix-facebook database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mautrix_facebook_base_path }}/mautrix-facebook.db {{ matrix_mautrix_facebook_data_path }}/mautrix-facebook.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mautrix_facebook_base_path }}/mautrix-facebook.db {{ matrix_mautrix_facebook_data_path }}/mautrix-facebook.db" + creates: "{{ matrix_mautrix_facebook_data_path }}/mautrix-facebook.db" + removes: "{{ matrix_mautrix_facebook_base_path }}/mautrix-facebook.db" when: "matrix_mautrix_facebook_stat_database.stat.exists" - name: Ensure mautrix-facebook config.yaml installed diff --git a/roles/custom/matrix-bridge-mautrix-googlechat/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-googlechat/tasks/main.yml index 917ba7a9..a23923b5 100644 --- a/roles/custom/matrix-bridge-mautrix-googlechat/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-googlechat/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_googlechat_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_googlechat_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-googlechat + - install-all + - install-mautrix-googlechat + block: - when: matrix_mautrix_googlechat_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_googlechat_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-googlechat - - install-all - - install-mautrix-googlechat - -- block: + block: - when: not matrix_mautrix_googlechat_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-googlechat diff --git a/roles/custom/matrix-bridge-mautrix-googlechat/tasks/setup_install.yml b/roles/custom/matrix-bridge-mautrix-googlechat/tasks/setup_install.yml index c1258471..c1ae8cf5 100644 --- a/roles/custom/matrix-bridge-mautrix-googlechat/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mautrix-googlechat/tasks/setup_install.yml @@ -91,7 +91,10 @@ when: "matrix_mautrix_googlechat_stat_database.stat.exists" - name: (Data relocation) Move mautrix-googlechat database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mautrix_googlechat_base_path }}/mautrix-googlechat.db {{ matrix_mautrix_googlechat_data_path }}/mautrix-googlechat.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mautrix_googlechat_base_path }}/mautrix-googlechat.db {{ matrix_mautrix_googlechat_data_path }}/mautrix-googlechat.db" + creates: "{{ matrix_mautrix_googlechat_data_path }}/mautrix-googlechat.db" + removes: "{{ matrix_mautrix_googlechat_base_path }}/mautrix-googlechat.db" when: "matrix_mautrix_googlechat_stat_database.stat.exists" - name: Ensure mautrix-googlechat config.yaml installed diff --git a/roles/custom/matrix-bridge-mautrix-hangouts/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-hangouts/tasks/main.yml index e9d8048c..e7881567 100644 --- a/roles/custom/matrix-bridge-mautrix-hangouts/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-hangouts/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_hangouts_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_hangouts_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-hangouts + - install-all + - install-mautrix-hangouts + block: - when: matrix_mautrix_hangouts_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_hangouts_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-hangouts - - install-all - - install-mautrix-hangouts - -- block: + block: - when: not matrix_mautrix_hangouts_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-hangouts diff --git a/roles/custom/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml b/roles/custom/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml index f8fdb023..0fa54994 100644 --- a/roles/custom/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml @@ -91,7 +91,10 @@ when: "matrix_mautrix_hangouts_stat_database.stat.exists" - name: (Data relocation) Move mautrix-hangouts database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mautrix_hangouts_base_path }}/mautrix-hangouts.db {{ matrix_mautrix_hangouts_data_path }}/mautrix-hangouts.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mautrix_hangouts_base_path }}/mautrix-hangouts.db {{ matrix_mautrix_hangouts_data_path }}/mautrix-hangouts.db" + creates: "{{ matrix_mautrix_hangouts_data_path }}/mautrix-hangouts.db" + removes: "{{ matrix_mautrix_hangouts_base_path }}/mautrix-hangouts.db" when: "matrix_mautrix_hangouts_stat_database.stat.exists" - name: Ensure mautrix-hangouts config.yaml installed diff --git a/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml index 0b3a722d..834dec0c 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_instagram_enabled | bool and matrix_mautrix_instagram_metrics_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_instagram_enabled | bool and matrix_mautrix_instagram_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-instagram + - install-all + - install-mautrix-instagram + block: - when: matrix_mautrix_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-instagram - - install-all - - install-mautrix-instagram - -- block: + block: - when: not matrix_mautrix_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-instagram diff --git a/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml index 9473344b..f5d769e5 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_signal_enabled | bool and matrix_mautrix_signal_metrics_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_signal_enabled | bool and matrix_mautrix_signal_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-signal + - install-all + - install-mautrix-signal + block: - when: matrix_mautrix_signal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_signal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-signal - - install-all - - install-mautrix-signal - -- block: + block: - when: not matrix_mautrix_signal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-signal diff --git a/roles/custom/matrix-bridge-mautrix-slack/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-slack/tasks/main.yml index bff90549..b7a2199d 100644 --- a/roles/custom/matrix-bridge-mautrix-slack/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-slack/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-mautrix-slack + - install-all + - install-mautrix-slack + block: - when: matrix_mautrix_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-slack - - install-all - - install-mautrix-slack - -- block: + block: - when: not matrix_mautrix_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-slack diff --git a/roles/custom/matrix-bridge-mautrix-telegram/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-telegram/tasks/main.yml index 3c8e6d04..212848b6 100644 --- a/roles/custom/matrix-bridge-mautrix-telegram/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-telegram/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_telegram_enabled | bool and matrix_mautrix_telegram_appservice_public_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_telegram_enabled | bool and matrix_mautrix_telegram_appservice_public_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-telegram + - install-all + - install-mautrix-telegram + block: - when: matrix_mautrix_telegram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_telegram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-telegram - - install-all - - install-mautrix-telegram - -- block: + block: - when: not matrix_mautrix_telegram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-telegram diff --git a/roles/custom/matrix-bridge-mautrix-telegram/tasks/setup_install.yml b/roles/custom/matrix-bridge-mautrix-telegram/tasks/setup_install.yml index 383d4827..ddf11d8d 100644 --- a/roles/custom/matrix-bridge-mautrix-telegram/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mautrix-telegram/tasks/setup_install.yml @@ -116,7 +116,10 @@ when: "matrix_mautrix_telegram_stat_database.stat.exists" - name: (Data relocation) Move mautrix-telegram database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mautrix_telegram_base_path }}/mautrix-telegram.db {{ matrix_mautrix_telegram_data_path }}/mautrix-telegram.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mautrix_telegram_base_path }}/mautrix-telegram.db {{ matrix_mautrix_telegram_data_path }}/mautrix-telegram.db" + creates: "{{ matrix_mautrix_telegram_data_path }}/mautrix-telegram.db" + removes: "{{ matrix_mautrix_telegram_base_path }}/mautrix-telegram.db" when: "matrix_mautrix_telegram_stat_database.stat.exists" - name: Ensure mautrix-telegram config.yaml installed diff --git a/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml index 9eb444a7..655e52fb 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_twitter_enabled | bool and matrix_mautrix_twitter_metrics_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_twitter_enabled | bool and matrix_mautrix_twitter_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-twitter + - install-all + - install-mautrix-twitter + block: - when: matrix_mautrix_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-twitter - - install-all - - install-mautrix-twitter - -- block: + block: - when: not matrix_mautrix_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-twitter diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml index b49a1300..cb4e8dea 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mautrix_whatsapp_enabled | bool and matrix_mautrix_whatsapp_metrics_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mautrix_whatsapp_enabled | bool and matrix_mautrix_whatsapp_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mautrix-whatsapp + - install-all + - install-mautrix-whatsapp + block: - when: matrix_mautrix_whatsapp_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mautrix_whatsapp_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mautrix-whatsapp - - install-all - - install-mautrix-whatsapp - -- block: + block: - when: not matrix_mautrix_whatsapp_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mautrix-whatsapp diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml index 1cf883c5..e41fe9c3 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml @@ -97,11 +97,17 @@ when: "matrix_mautrix_whatsapp_stat_database.stat.exists" - name: (Data relocation) Move mautrix-whatsapp database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mautrix_whatsapp_base_path }}/mautrix-whatsapp.db {{ matrix_mautrix_whatsapp_data_path }}/mautrix-whatsapp.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mautrix_whatsapp_base_path }}/mautrix-whatsapp.db {{ matrix_mautrix_whatsapp_data_path }}/mautrix-whatsapp.db" + creates: "{{ matrix_mautrix_whatsapp_data_path }}/mautrix-whatsapp.db" + removes: "{{ matrix_mautrix_whatsapp_base_path }}/mautrix-whatsapp.db" when: "matrix_mautrix_whatsapp_stat_database.stat.exists" - name: (Data relocation) Move mautrix-whatsapp mx-state file to ./data directory - ansible.builtin.command: "mv {{ matrix_mautrix_whatsapp_base_path }}/mx-state.json {{ matrix_mautrix_whatsapp_data_path }}/mx-state.json" + ansible.builtin.command: + cmd: "mv {{ matrix_mautrix_whatsapp_base_path }}/mx-state.json {{ matrix_mautrix_whatsapp_data_path }}/mx-state.json" + creates: "{{ matrix_mautrix_whatsapp_data_path }}/mx-state.json" + removes: "{{ matrix_mautrix_whatsapp_base_path }}/mx-state.json" when: "matrix_mautrix_whatsapp_stat_mx_state.stat.exists" - name: Ensure mautrix-whatsapp config.yaml installed diff --git a/roles/custom/matrix-bridge-mx-puppet-discord/tasks/main.yml b/roles/custom/matrix-bridge-mx-puppet-discord/tasks/main.yml index c1403dfa..c0300b51 100644 --- a/roles/custom/matrix-bridge-mx-puppet-discord/tasks/main.yml +++ b/roles/custom/matrix-bridge-mx-puppet-discord/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-mx-puppet-discord + - install-all + - install-mx-puppet-discord + block: - when: matrix_mx_puppet_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mx_puppet_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mx-puppet-discord - - install-all - - install-mx-puppet-discord - -- block: + block: - when: not matrix_mx_puppet_discord_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mx-puppet-discord diff --git a/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/main.yml b/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/main.yml index ab5b0d44..de225e71 100644 --- a/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/main.yml +++ b/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-mx-puppet-groupme + - install-all + - install-mx-puppet-groupme + block: - when: matrix_mx_puppet_groupme_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mx_puppet_groupme_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mx-puppet-groupme - - install-all - - install-mx-puppet-groupme - -- block: + block: - when: not matrix_mx_puppet_groupme_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mx-puppet-groupme diff --git a/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/setup_install.yml b/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/setup_install.yml index 9db25e86..0ba1e699 100644 --- a/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mx-puppet-groupme/tasks/setup_install.yml @@ -29,7 +29,10 @@ when: "matrix_mx_puppet_groupme_stat_database.stat.exists" - name: (Data relocation) Move mx-puppet-groupme database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mx_puppet_groupme_base_path }}/database.db {{ matrix_mx_puppet_groupme_data_path }}/database.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mx_puppet_groupme_base_path }}/database.db {{ matrix_mx_puppet_groupme_data_path }}/database.db" + creates: "{{ matrix_mx_puppet_groupme_data_path }}/database.db" + removes: "{{ matrix_mx_puppet_groupme_base_path }}/database.db" when: "matrix_mx_puppet_groupme_stat_database.stat.exists" - ansible.builtin.set_fact: diff --git a/roles/custom/matrix-bridge-mx-puppet-instagram/tasks/main.yml b/roles/custom/matrix-bridge-mx-puppet-instagram/tasks/main.yml index 3cf02773..0d46532c 100644 --- a/roles/custom/matrix-bridge-mx-puppet-instagram/tasks/main.yml +++ b/roles/custom/matrix-bridge-mx-puppet-instagram/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-mx-puppet-instagram + - install-all + - install-mx-puppet-instagram + block: - when: matrix_mx_puppet_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mx_puppet_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mx-puppet-instagram - - install-all - - install-mx-puppet-instagram - -- block: + block: - when: not matrix_mx_puppet_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mx-puppet-instagram diff --git a/roles/custom/matrix-bridge-mx-puppet-slack/tasks/main.yml b/roles/custom/matrix-bridge-mx-puppet-slack/tasks/main.yml index 3d6e722b..63e07405 100644 --- a/roles/custom/matrix-bridge-mx-puppet-slack/tasks/main.yml +++ b/roles/custom/matrix-bridge-mx-puppet-slack/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mx_puppet_slack_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mx_puppet_slack_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mx-puppet-slack + - install-all + - install-mx-puppet-slack + block: - when: matrix_mx_puppet_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mx_puppet_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mx-puppet-slack - - install-all - - install-mx-puppet-slack - -- block: + block: - when: not matrix_mx_puppet_slack_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mx-puppet-slack diff --git a/roles/custom/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml b/roles/custom/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml index e75d47e5..bba2d25c 100644 --- a/roles/custom/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml @@ -91,7 +91,10 @@ when: "matrix_mx_puppet_slack_enabled | bool and matrix_mx_puppet_slack_container_image_self_build" - name: (Data relocation) Move mx-puppet-slack database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mx_puppet_slack_base_path }}/database.db {{ matrix_mx_puppet_slack_data_path }}/database.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mx_puppet_slack_base_path }}/database.db {{ matrix_mx_puppet_slack_data_path }}/database.db" + creates: "{{ matrix_mx_puppet_slack_data_path }}/database.db" + removes: "{{ matrix_mx_puppet_slack_base_path }}/database.db" when: "matrix_mx_puppet_slack_stat_database.stat.exists" - name: Ensure mx-puppet-slack config.yaml installed diff --git a/roles/custom/matrix-bridge-mx-puppet-steam/tasks/main.yml b/roles/custom/matrix-bridge-mx-puppet-steam/tasks/main.yml index 6eacc6fc..66d2980d 100644 --- a/roles/custom/matrix-bridge-mx-puppet-steam/tasks/main.yml +++ b/roles/custom/matrix-bridge-mx-puppet-steam/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-mx-puppet-steam + - install-all + - install-mx-puppet-steam + block: - when: matrix_mx_puppet_steam_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mx_puppet_steam_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mx-puppet-steam - - install-all - - install-mx-puppet-steam - -- block: + block: - when: not matrix_mx_puppet_steam_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mx-puppet-steam diff --git a/roles/custom/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml b/roles/custom/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml index ca2de448..f2ccea59 100644 --- a/roles/custom/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml @@ -29,7 +29,10 @@ when: "matrix_mx_puppet_steam_stat_database.stat.exists" - name: (Data relocation) Move mx-puppet-steam database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mx_puppet_steam_base_path }}/database.db {{ matrix_mx_puppet_steam_data_path }}/database.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mx_puppet_steam_base_path }}/database.db {{ matrix_mx_puppet_steam_data_path }}/database.db" + creates: "{{ matrix_mx_puppet_steam_data_path }}/database.db" + removes: "{{ matrix_mx_puppet_steam_base_path }}/database.db" when: "matrix_mx_puppet_steam_stat_database.stat.exists" - ansible.builtin.set_fact: diff --git a/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/main.yml b/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/main.yml index 85c06f04..d6d68103 100644 --- a/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/main.yml +++ b/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_mx_puppet_twitter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_mx_puppet_twitter_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-mx-puppet-twitter + - install-all + - install-mx-puppet-twitter + block: - when: matrix_mx_puppet_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_mx_puppet_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-mx-puppet-twitter - - install-all - - install-mx-puppet-twitter - -- block: + block: - when: not matrix_mx_puppet_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-mx-puppet-twitter diff --git a/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml b/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml index a8450205..9224e956 100644 --- a/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml @@ -29,7 +29,10 @@ when: "matrix_mx_puppet_twitter_stat_database.stat.exists" - name: (Data relocation) Move mx-puppet-twitter database file to ./data directory - ansible.builtin.command: "mv {{ matrix_mx_puppet_twitter_base_path }}/database.db {{ matrix_mx_puppet_twitter_data_path }}/database.db" + ansible.builtin.command: + cmd: "mv {{ matrix_mx_puppet_twitter_base_path }}/database.db {{ matrix_mx_puppet_twitter_data_path }}/database.db" + creates: "{{ matrix_mx_puppet_twitter_data_path }}/database.db" + removes: "{{ matrix_mx_puppet_twitter_base_path }}/database.db" when: "matrix_mx_puppet_twitter_stat_database.stat.exists" - ansible.builtin.set_fact: diff --git a/roles/custom/matrix-bridge-sms/tasks/main.yml b/roles/custom/matrix-bridge-sms/tasks/main.yml index f9bd502c..57d51727 100644 --- a/roles/custom/matrix-bridge-sms/tasks/main.yml +++ b/roles/custom/matrix-bridge-sms/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-matrix-sms-bridge + - install-all + - install-matrix-sms-bridge + block: - when: matrix_sms_bridge_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_sms_bridge_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-matrix-sms-bridge - - install-all - - install-matrix-sms-bridge - -- block: + block: - when: not matrix_sms_bridge_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-matrix-sms-bridge diff --git a/roles/custom/matrix-cactus-comments/tasks/main.yml b/roles/custom/matrix-cactus-comments/tasks/main.yml index c6108e91..c1a08d32 100644 --- a/roles/custom/matrix-cactus-comments/tasks/main.yml +++ b/roles/custom/matrix-cactus-comments/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_cactus_comments_enabled | bool and matrix_cactus_comments_serve_client_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_cactus_comments_enabled | bool and matrix_cactus_comments_serve_client_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-cactus-comments + - install-all + - install-cactus-comments + block: - when: matrix_cactus_comments_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_cactus_comments_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-cactus-comments - - install-all - - install-cactus-comments - -- block: + block: - when: not matrix_cactus_comments_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-cactus-comments diff --git a/roles/custom/matrix-client-cinny/tasks/main.yml b/roles/custom/matrix-client-cinny/tasks/main.yml index 1e6d4759..40c6514b 100644 --- a/roles/custom/matrix-client-cinny/tasks/main.yml +++ b/roles/custom/matrix-client-cinny/tasks/main.yml @@ -1,26 +1,26 @@ --- -- block: +- tags: + - setup-all + - setup-client-cinny + - install-all + - install-client-cinny + block: - when: matrix_client_cinny_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_client_cinny_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-client-cinny - - install-all - - install-client-cinny - -- block: + block: - when: not matrix_client_cinny_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-client-cinny -- block: +- tags: + - self-check + block: - when: matrix_client_cinny_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check.yml" - tags: - - self-check diff --git a/roles/custom/matrix-client-element/tasks/main.yml b/roles/custom/matrix-client-element/tasks/main.yml index 89e50ba7..77020067 100644 --- a/roles/custom/matrix-client-element/tasks/main.yml +++ b/roles/custom/matrix-client-element/tasks/main.yml @@ -1,6 +1,11 @@ --- -- block: +- tags: + - setup-all + - setup-client-element + - install-all + - install-client-element + block: - when: matrix_client_element_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" @@ -9,21 +14,16 @@ - when: matrix_client_element_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-client-element - - install-all - - install-client-element - -- block: + block: - when: not matrix_client_element_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-client-element -- block: +- tags: + - self-check + block: - when: matrix_client_element_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check.yml" - tags: - - self-check diff --git a/roles/custom/matrix-client-element/tasks/setup_install.yml b/roles/custom/matrix-client-element/tasks/setup_install.yml index fc0532db..50d6370f 100644 --- a/roles/custom/matrix-client-element/tasks/setup_install.yml +++ b/roles/custom/matrix-client-element/tasks/setup_install.yml @@ -56,6 +56,7 @@ --tag={{ matrix_client_element_docker_image }} --file={{ matrix_client_element_docker_src_files_path }}/Dockerfile {{ matrix_client_element_docker_src_files_path }} + changed_when: true when: matrix_client_element_container_image_self_build | bool - name: Ensure Element configuration installed diff --git a/roles/custom/matrix-client-hydrogen/tasks/main.yml b/roles/custom/matrix-client-hydrogen/tasks/main.yml index 9bb8bc08..9e7445c0 100644 --- a/roles/custom/matrix-client-hydrogen/tasks/main.yml +++ b/roles/custom/matrix-client-hydrogen/tasks/main.yml @@ -1,26 +1,26 @@ --- -- block: +- tags: + - setup-all + - setup-client-hydrogen + - install-all + - install-client-hydrogen + block: - when: matrix_client_hydrogen_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_client_hydrogen_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-client-hydrogen - - install-all - - install-client-hydrogen - -- block: + block: - when: not matrix_client_hydrogen_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-client-hydrogen -- block: +- tags: + - self-check + block: - when: matrix_client_hydrogen_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check.yml" - tags: - - self-check diff --git a/roles/custom/matrix-common-after/tasks/main.yml b/roles/custom/matrix-common-after/tasks/main.yml index fe23e467..5e4dc76b 100644 --- a/roles/custom/matrix-common-after/tasks/main.yml +++ b/roles/custom/matrix-common-after/tasks/main.yml @@ -1,6 +1,6 @@ --- -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/run_docker_prune.yml" - tags: +- tags: - run-docker-prune + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/run_docker_prune.yml" diff --git a/roles/custom/matrix-conduit/tasks/main.yml b/roles/custom/matrix-conduit/tasks/main.yml index d733ac08..f48e385e 100644 --- a/roles/custom/matrix-conduit/tasks/main.yml +++ b/roles/custom/matrix-conduit/tasks/main.yml @@ -1,17 +1,17 @@ --- -- block: - - when: matrix_conduit_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: +- tags: - setup-all - setup-conduit - install-all - install-conduit + block: + - when: matrix_conduit_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" -- block: - - when: not matrix_conduit_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-conduit + block: + - when: not matrix_conduit_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" diff --git a/roles/custom/matrix-corporal/tasks/main.yml b/roles/custom/matrix-corporal/tasks/main.yml index c4c7b7fc..e8f3b785 100644 --- a/roles/custom/matrix-corporal/tasks/main.yml +++ b/roles/custom/matrix-corporal/tasks/main.yml @@ -1,26 +1,26 @@ --- -- block: +- tags: + - setup-all + - setup-corporal + - install-all + - install-corporal + block: - when: matrix_corporal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_corporal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-corporal - - install-all - - install-corporal - -- block: + block: - when: not matrix_corporal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-corporal -- block: +- tags: + - self-check + block: - when: matrix_corporal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check.yml" - tags: - - self-check diff --git a/roles/custom/matrix-coturn/tasks/main.yml b/roles/custom/matrix-coturn/tasks/main.yml index 45ce9fd9..1b1332fb 100644 --- a/roles/custom/matrix-coturn/tasks/main.yml +++ b/roles/custom/matrix-coturn/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-coturn + - install-all + - install-coturn + block: - when: matrix_coturn_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_coturn_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-coturn - - install-all - - install-coturn - -- block: + block: - when: not matrix_coturn_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-coturn diff --git a/roles/custom/matrix-coturn/tasks/setup_install.yml b/roles/custom/matrix-coturn/tasks/setup_install.yml index 44325f46..1cb8bd69 100644 --- a/roles/custom/matrix-coturn/tasks/setup_install.yml +++ b/roles/custom/matrix-coturn/tasks/setup_install.yml @@ -62,8 +62,8 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" -- when: matrix_coturn_docker_network not in ['', 'host'] - name: Ensure Coturn network is created in Docker +- name: Ensure Coturn network is created in Docker + when: matrix_coturn_docker_network not in ['', 'host'] community.docker.docker_network: name: "{{ matrix_coturn_docker_network }}" driver: bridge diff --git a/roles/custom/matrix-dendrite/tasks/main.yml b/roles/custom/matrix-dendrite/tasks/main.yml index 0e88ec51..78d81524 100644 --- a/roles/custom/matrix-dendrite/tasks/main.yml +++ b/roles/custom/matrix-dendrite/tasks/main.yml @@ -1,35 +1,35 @@ --- -- block: +- tags: + - setup-all + - setup-dendrite + - install-all + - install-dendrite + block: - when: matrix_dendrite_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_dendrite_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-dendrite - - install-all - - install-dendrite - -- block: + block: - when: not matrix_dendrite_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-dendrite -- block: +- tags: + - self-check + block: - when: matrix_dendrite_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_client_api.yml" - when: matrix_dendrite_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_federation_api.yml" - tags: - - self-check -- block: +- tags: + - register-user + block: - when: matrix_dendrite_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/register_user.yml" - tags: - - register-user diff --git a/roles/custom/matrix-dendrite/tasks/setup_install.yml b/roles/custom/matrix-dendrite/tasks/setup_install.yml index 3b8cbd9b..283e8b01 100644 --- a/roles/custom/matrix-dendrite/tasks/setup_install.yml +++ b/roles/custom/matrix-dendrite/tasks/setup_install.yml @@ -42,24 +42,20 @@ delay: "{{ devture_playbook_help_container_retries_delay }}" until: result is not failed -- name: Check if a Dendrite signing key exists - ansible.builtin.stat: - path: "{{ matrix_dendrite_config_dir_path }}/{{ matrix_server_fqn_matrix }}.signing.pem" - register: matrix_dendrite_signing_key_stat - # We do this so that the signing key would get generated. # We don't use the `docker_container` module, because using it with `cap_drop` requires # a very recent version, which is not available for a lot of people yet. - name: Generate Dendrite signing key - ansible.builtin.command: | - docker run - --rm - --name=matrix-dendrite-config - --entrypoint=generate-keys - --mount type=bind,src={{ matrix_dendrite_config_dir_path }},dst=/data - {{ matrix_dendrite_docker_image }} --private-key=/data/{{ matrix_server_fqn_matrix }}.signing.pem - generate - when: "not matrix_dendrite_signing_key_stat.stat.exists" + ansible.builtin.command: + cmd: | + docker run + --rm + --name=matrix-dendrite-config + --entrypoint=generate-keys + --mount type=bind,src={{ matrix_dendrite_config_dir_path }},dst=/data + {{ matrix_dendrite_docker_image }} --private-key=/data/{{ matrix_server_fqn_matrix }}.signing.pem + generate + creates: "{{ matrix_dendrite_config_dir_path }}/{{ matrix_server_fqn_matrix }}.signing.pem" - name: Ensure Dendrite server key exists ansible.builtin.file: diff --git a/roles/custom/matrix-dimension/tasks/main.yml b/roles/custom/matrix-dimension/tasks/main.yml index efd0f8d4..1e949d3f 100644 --- a/roles/custom/matrix-dimension/tasks/main.yml +++ b/roles/custom/matrix-dimension/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-dimension + - install-all + - install-dimension + block: - when: matrix_dimension_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_dimension_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-dimension - - install-all - - install-dimension - -- block: + block: - when: not matrix_dimension_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-dimension diff --git a/roles/custom/matrix-dynamic-dns/tasks/main.yml b/roles/custom/matrix-dynamic-dns/tasks/main.yml index ae7503b7..8327a84a 100644 --- a/roles/custom/matrix-dynamic-dns/tasks/main.yml +++ b/roles/custom/matrix-dynamic-dns/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-dynamic-dns + - install-all + - install-dynamic-dns + block: - when: matrix_dynamic_dns_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_dynamic_dns_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-dynamic-dns - - install-all - - install-dynamic-dns - -- block: + block: - when: not matrix_dynamic_dns_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-dynamic-dns diff --git a/roles/custom/matrix-email2matrix/tasks/main.yml b/roles/custom/matrix-email2matrix/tasks/main.yml index 8fac77ce..acd2c9c1 100644 --- a/roles/custom/matrix-email2matrix/tasks/main.yml +++ b/roles/custom/matrix-email2matrix/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-email2matrix + - install-all + - install-email2matrix + block: - when: matrix_email2matrix_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_email2matrix_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-email2matrix - - install-all - - install-email2matrix - -- block: + block: - when: not matrix_email2matrix_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-email2matrix diff --git a/roles/custom/matrix-jitsi/tasks/main.yml b/roles/custom/matrix-jitsi/tasks/main.yml index b35c2c34..0ebf3ad6 100644 --- a/roles/custom/matrix-jitsi/tasks/main.yml +++ b/roles/custom/matrix-jitsi/tasks/main.yml @@ -1,27 +1,32 @@ --- -- block: +- tags: + - setup-additional-jitsi-jvb + - install-additional-jitsi-jvb + block: - when: matrix_jitsi_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/init_additional_jvb.yml" - tags: + +- tags: + - setup-all + - setup-jitsi - setup-additional-jitsi-jvb + - install-all + - install-jitsi - install-additional-jitsi-jvb - -- block: + block: - when: matrix_jitsi_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_jitsi_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_base.yml" - tags: + +- tags: - setup-all - setup-jitsi - - setup-additional-jitsi-jvb - install-all - install-jitsi - - install-additional-jitsi-jvb - -- block: + block: - when: matrix_jitsi_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_web_install.yml" @@ -30,13 +35,11 @@ - when: matrix_jitsi_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_jicofo_install.yml" - tags: + +- tags: - setup-all - setup-jitsi - - install-all - - install-jitsi - -- block: + block: - when: not matrix_jitsi_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_web_uninstall.yml" @@ -45,31 +48,28 @@ - when: not matrix_jitsi_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_jicofo_uninstall.yml" - tags: - - setup-all - - setup-jitsi -- block: - - when: matrix_jitsi_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_jvb_install.yml" - tags: +- tags: - setup-all - setup-jitsi - setup-additional-jitsi-jvb - install-all - install-jitsi - install-additional-jitsi-jvb + block: + - when: matrix_jitsi_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_jvb_install.yml" -- block: - - when: not matrix_jitsi_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_jvb_uninstall.yml" - tags: +- tags: - setup-all - setup-jitsi - setup-additional-jitsi-jvb + block: + - when: not matrix_jitsi_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_jitsi_jvb_uninstall.yml" -- block: +- tags: + - self-check + block: - when: matrix_jitsi_enable_auth | bool and matrix_jitsi_auth_type == 'matrix' ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_matrix_auth.yml" - tags: - - self-check diff --git a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml index f1d9ff21..4aa67a31 100644 --- a/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml +++ b/roles/custom/matrix-jitsi/tasks/util/setup_jitsi_auth_uvs_install.yml @@ -5,11 +5,16 @@ repo: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_location }}" dest: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_target }}" version: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_version }}" + become: true + become_user: "{{ matrix_user_username }}" - name: Install Prosody Auth Matrix User Verification Plugin ansible.builtin.copy: remote_src: true src: "{{ matrix_jitsi_prosody_auth_matrix_user_verification_repo_target }}/{{ item.path }}" dest: "{{ matrix_jitsi_prosody_plugins_path }}/{{ item.path }}" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" with_items: "{{ matrix_jitsi_prosody_auth_matrix_files }}" when: item.when | bool diff --git a/roles/custom/matrix-ldap-registration-proxy/tasks/main.yml b/roles/custom/matrix-ldap-registration-proxy/tasks/main.yml index 9309113f..cc71ae82 100644 --- a/roles/custom/matrix-ldap-registration-proxy/tasks/main.yml +++ b/roles/custom/matrix-ldap-registration-proxy/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_ldap_registration_proxy_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_ldap_registration_proxy_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-matrix-ldap-registration-proxy + - install-all + - install-matrix-ldap-registration-proxy + block: - when: matrix_ldap_registration_proxy_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_ldap_registration_proxy_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-matrix-ldap-registration-proxy - - install-all - - install-matrix-ldap-registration-proxy - -- block: + block: - when: not matrix_ldap_registration_proxy_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-matrix-ldap-registration-proxy diff --git a/roles/custom/matrix-ma1sd/tasks/main.yml b/roles/custom/matrix-ma1sd/tasks/main.yml index a65cf371..a9c4ec8b 100644 --- a/roles/custom/matrix-ma1sd/tasks/main.yml +++ b/roles/custom/matrix-ma1sd/tasks/main.yml @@ -1,25 +1,25 @@ --- -- block: +- tags: + - setup-all + - setup-ma1sd + - install-all + - install-ma1sd + block: - when: matrix_ma1sd_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_ma1sd_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-ma1sd - - install-all - - install-ma1sd -- block: +- tags: + - setup-ma1sd + block: - when: not matrix_ma1sd_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-ma1sd -- block: +- tags: + - self-check + block: - when: matrix_ma1sd_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check.yml" - tags: - - self-check diff --git a/roles/custom/matrix-ma1sd/tasks/setup_install.yml b/roles/custom/matrix-ma1sd/tasks/setup_install.yml index 6fe20342..5800b45a 100644 --- a/roles/custom/matrix-ma1sd/tasks/setup_install.yml +++ b/roles/custom/matrix-ma1sd/tasks/setup_install.yml @@ -89,6 +89,7 @@ chdir: "{{ matrix_ma1sd_docker_src_files_path }}" environment: DOCKER_BUILDKIT: 1 + changed_when: true when: matrix_ma1sd_git_pull_results.changed - name: Ensure ma1sd Docker image is tagged correctly diff --git a/roles/custom/matrix-mailer/tasks/main.yml b/roles/custom/matrix-mailer/tasks/main.yml index b269f63e..00fa0884 100644 --- a/roles/custom/matrix-mailer/tasks/main.yml +++ b/roles/custom/matrix-mailer/tasks/main.yml @@ -1,17 +1,17 @@ --- -- block: - - when: matrix_mailer_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: +- tags: - setup-all - setup-mailer - install-all - install-mailer + block: + - when: matrix_mailer_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" -- block: - - when: not matrix_mailer_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-mailer + block: + - when: not matrix_mailer_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" diff --git a/roles/custom/matrix-nginx-proxy/tasks/main.yml b/roles/custom/matrix-nginx-proxy/tasks/main.yml index 39cec7e5..7a702fb3 100644 --- a/roles/custom/matrix-nginx-proxy/tasks/main.yml +++ b/roles/custom/matrix-nginx-proxy/tasks/main.yml @@ -29,13 +29,13 @@ - install-all - install-nginx-proxy -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_well_known.yml" - tags: +- tags: - self-check + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_well_known.yml" - name: Mark matrix-nginx-proxy role as executed - ansible.builtin.set_fact: - matrix_nginx_proxy_role_executed: true tags: - always + ansible.builtin.set_fact: + matrix_nginx_proxy_role_executed: true diff --git a/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt_obtain_for_domain.yml b/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt_obtain_for_domain.yml index b10791a7..a175b831 100644 --- a/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt_obtain_for_domain.yml +++ b/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt_obtain_for_domain.yml @@ -51,6 +51,7 @@ --agree-tos --email={{ matrix_ssl_lets_encrypt_support_email }} -d {{ domain_name }} + changed_when: true when: domain_name_needs_cert | bool register: result_certbot_direct ignore_errors: true @@ -81,6 +82,7 @@ --agree-tos --email={{ matrix_ssl_lets_encrypt_support_email }} -d {{ domain_name }} + changed_when: true when: "domain_name_needs_cert and result_certbot_direct.failed" register: result_certbot_proxy ignore_errors: true diff --git a/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_self_signed_obtain_for_domain.yml b/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_self_signed_obtain_for_domain.yml index d0b254a2..c0f19519 100644 --- a/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_self_signed_obtain_for_domain.yml +++ b/roles/custom/matrix-nginx-proxy/tasks/ssl/setup_ssl_self_signed_obtain_for_domain.yml @@ -28,16 +28,19 @@ # # We'll do it in a more manual way. - name: Generate SSL certificate - ansible.builtin.command: | - openssl req -x509 \ - -sha256 \ - -newkey rsa:4096 \ - -nodes \ - -subj "/CN={{ domain_name }}" \ - -keyout {{ matrix_ssl_certificate_cert_key_path }} \ - -out {{ matrix_ssl_certificate_cert_path }} \ - -days 3650 when: "not matrix_ssl_certificate_cert_path_stat_result.stat.exists" + ansible.builtin.command: + cmd: | + openssl req -x509 \ + -sha256 \ + -newkey rsa:4096 \ + -nodes \ + -subj "/CN={{ domain_name }}" \ + -keyout {{ matrix_ssl_certificate_cert_key_path }} \ + -out {{ matrix_ssl_certificate_cert_path }} \ + -days 3650 + # Well, this creates 2 files, but Ansible can only check 1. + creates: "{{ matrix_ssl_certificate_cert_path }}" - name: Adjust SSL certificate file ownership ansible.builtin.file: diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/main.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/main.yml index b205dec3..ad047460 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/main.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: +- tags: + - setup-all + - setup-prometheus-nginxlog-exporter + - install-all + - install-prometheus-nginxlog-exporter + block: - when: matrix_prometheus_nginxlog_exporter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: + +- tags: - setup-all - setup-prometheus-nginxlog-exporter - install-all - install-prometheus-nginxlog-exporter - -- block: + block: - when: matrix_prometheus_nginxlog_exporter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_prometheus_nginxlog_exporter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-prometheus-nginxlog-exporter - - install-all - - install-prometheus-nginxlog-exporter - -- block: + block: - when: not matrix_prometheus_nginxlog_exporter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-prometheus-nginxlog-exporter diff --git a/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml index e5654688..ea067bdb 100644 --- a/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml @@ -1,28 +1,28 @@ --- - name: Fail if matrix-nginx-proxy role executed before matrix-prometheus-services-proxy-connect + when: matrix_nginx_proxy_role_executed | default(False) | bool ansible.builtin.fail: msg: >- Trying to append node-exporter's reverse-proxying configuration to matrix-nginx-proxy, but it's pointless since the matrix-nginx-proxy role had already executed. To fix this, please change the order of roles in your playbook, so that the matrix-nginx-proxy role would run after the matrix-prometheus-services-proxy-connect role. - when: matrix_nginx_proxy_role_executed | default(False) | bool -- block: - - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml" -- block: - - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml" diff --git a/roles/custom/matrix-prometheus/tasks/main.yml b/roles/custom/matrix-prometheus/tasks/main.yml index 59371d4d..bbd82001 100644 --- a/roles/custom/matrix-prometheus/tasks/main.yml +++ b/roles/custom/matrix-prometheus/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-prometheus + - install-all + - install-prometheus + block: - when: matrix_prometheus_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_prometheus_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-prometheus - - install-all - - install-prometheus - -- block: + block: - when: not matrix_prometheus_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-prometheus diff --git a/roles/custom/matrix-rageshake/tasks/install.yml b/roles/custom/matrix-rageshake/tasks/install.yml index f9461680..a1db9a87 100644 --- a/roles/custom/matrix-rageshake/tasks/install.yml +++ b/roles/custom/matrix-rageshake/tasks/install.yml @@ -62,6 +62,7 @@ --tag={{ matrix_rageshake_container_image }} --file={{ matrix_rageshake_container_src_files_path }}/Dockerfile {{ matrix_rageshake_container_src_files_path }} + changed_when: true when: matrix_rageshake_container_image_self_build | bool - name: Ensure rageshake container network is created diff --git a/roles/custom/matrix-rageshake/tasks/main.yml b/roles/custom/matrix-rageshake/tasks/main.yml index c330964a..a211296d 100644 --- a/roles/custom/matrix-rageshake/tasks/main.yml +++ b/roles/custom/matrix-rageshake/tasks/main.yml @@ -1,19 +1,20 @@ --- -- block: + +- tags: + - setup-all + - setup-rageshake + - install-all + - install-rageshake + block: - when: matrix_rageshake_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_rageshake_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" - tags: + +- tags: - setup-all - setup-rageshake - - install-all - - install-rageshake - -- block: + block: - when: not matrix_rageshake_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" - tags: - - setup-all - - setup-rageshake diff --git a/roles/custom/matrix-registration/tasks/main.yml b/roles/custom/matrix-registration/tasks/main.yml index bebcebc0..64843dee 100644 --- a/roles/custom/matrix-registration/tasks/main.yml +++ b/roles/custom/matrix-registration/tasks/main.yml @@ -1,41 +1,41 @@ --- -- block: - - when: matrix_registration_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_registration_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-matrix-registration + - install-all + - install-matrix-registration + block: - when: matrix_registration_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_registration_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-matrix-registration - - install-all - - install-matrix-registration - -- block: + block: - when: not matrix_registration_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-matrix-registration -- block: +- tags: + - generate-matrix-registration-token + block: - when: matrix_registration_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/generate_token.yml" - tags: - - generate-matrix-registration-token -- block: +- tags: + - list-matrix-registration-tokens + block: - when: matrix_registration_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/list_tokens.yml" - tags: - - list-matrix-registration-tokens diff --git a/roles/custom/matrix-sliding-sync/tasks/install.yml b/roles/custom/matrix-sliding-sync/tasks/install.yml index 26717afd..cb41cec0 100644 --- a/roles/custom/matrix-sliding-sync/tasks/install.yml +++ b/roles/custom/matrix-sliding-sync/tasks/install.yml @@ -56,6 +56,7 @@ --tag={{ matrix_sliding_sync_container_image }} --file={{ matrix_sliding_sync_container_src_path }}/Dockerfile {{ matrix_sliding_sync_container_src_path }} + changed_when: true - name: Ensure matrix-sliding-sync container network is created community.general.docker_network: diff --git a/roles/custom/matrix-sliding-sync/tasks/main.yml b/roles/custom/matrix-sliding-sync/tasks/main.yml index 71e7391b..521f16fe 100644 --- a/roles/custom/matrix-sliding-sync/tasks/main.yml +++ b/roles/custom/matrix-sliding-sync/tasks/main.yml @@ -1,19 +1,20 @@ --- -- block: + +- tags: + - setup-all + - setup-sliding-sync + - install-all + - install-sliding-sync + block: - when: matrix_sliding_sync_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_sliding_sync_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" - tags: + +- tags: - setup-all - setup-sliding-sync - - install-all - - install-sliding-sync - -- block: + block: - when: not matrix_sliding_sync_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" - tags: - - setup-all - - setup-sliding-sync diff --git a/roles/custom/matrix-sygnal/tasks/main.yml b/roles/custom/matrix-sygnal/tasks/main.yml index ba4cb876..a586cbd0 100644 --- a/roles/custom/matrix-sygnal/tasks/main.yml +++ b/roles/custom/matrix-sygnal/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: +- tags: + - setup-all + - setup-sygnal + - install-all + - install-sygnal + block: - when: matrix_sygnal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_sygnal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" - tags: + +- tags: - setup-all - setup-sygnal - - install-all - - install-sygnal - -- block: + block: - when: not matrix_sygnal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" - tags: - - setup-all - - setup-sygnal diff --git a/roles/custom/matrix-synapse-admin/tasks/main.yml b/roles/custom/matrix-synapse-admin/tasks/main.yml index f38f1d5e..da9bdb18 100644 --- a/roles/custom/matrix-synapse-admin/tasks/main.yml +++ b/roles/custom/matrix-synapse-admin/tasks/main.yml @@ -1,29 +1,29 @@ --- -- block: - - when: matrix_synapse_admin_enabled | bool and matrix_synapse_admin_nginx_proxy_integration_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: +- tags: - setup-all - setup-nginx-proxy - install-all - install-nginx-proxy + block: + - when: matrix_synapse_admin_enabled | bool and matrix_synapse_admin_nginx_proxy_integration_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" -- block: +- tags: + - setup-all + - setup-synapse-admin + - install-all + - install-synapse-admin + block: - when: matrix_synapse_admin_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_synapse_admin_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-synapse-admin - - install-all - - install-synapse-admin - -- block: + block: - when: not matrix_synapse_admin_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-synapse-admin diff --git a/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/main.yml b/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/main.yml index 0dba0b9c..68060267 100644 --- a/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/main.yml +++ b/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/main.yml @@ -1,20 +1,20 @@ --- -- block: - - when: matrix_synapse_reverse_proxy_companion_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: +- tags: - setup-all - setup-synapse-reverse-proxy-companion - setup-synapse - install-all - install-synapse-reverse-proxy-companion - install-synapse + block: + - when: matrix_synapse_reverse_proxy_companion_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" -- block: - - when: not matrix_synapse_reverse_proxy_companion_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse-reverse-proxy-companion - setup-synapse + block: + - when: not matrix_synapse_reverse_proxy_companion_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" diff --git a/roles/custom/matrix-synapse/tasks/ext/setup_install.yml b/roles/custom/matrix-synapse/tasks/ext/setup_install.yml index a2413803..0d887939 100644 --- a/roles/custom/matrix-synapse/tasks/ext/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/ext/setup_install.yml @@ -1,60 +1,60 @@ --- # encryption-disabler -- block: - - when: matrix_synapse_ext_encryption_disabler_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/encryption-disabler/setup_install.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: matrix_synapse_ext_encryption_disabler_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/encryption-disabler/setup_install.yml" # rest-auth -- block: - - when: matrix_synapse_ext_password_provider_rest_auth_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/rest-auth/setup_install.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: matrix_synapse_ext_password_provider_rest_auth_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/rest-auth/setup_install.yml" # shared-secret-auth -- block: - - when: matrix_synapse_ext_password_provider_shared_secret_auth_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/shared-secret-auth/setup_install.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: matrix_synapse_ext_password_provider_shared_secret_auth_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/shared-secret-auth/setup_install.yml" # ldap-auth -- block: - - when: matrix_synapse_ext_password_provider_ldap_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/ldap-auth/setup_install.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: matrix_synapse_ext_password_provider_ldap_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/ldap-auth/setup_install.yml" # synapse-simple-antispam -- block: - - when: matrix_synapse_ext_spam_checker_synapse_simple_antispam_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/synapse-simple-antispam/setup_install.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: matrix_synapse_ext_spam_checker_synapse_simple_antispam_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/synapse-simple-antispam/setup_install.yml" # mjolnir-antispam -- block: - - when: matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/mjolnir-antispam/setup_install.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/mjolnir-antispam/setup_install.yml" # s3-storage-provider -- block: +- tags: + - setup-all + - setup-synapse + block: - when: matrix_synapse_ext_synapse_s3_storage_provider_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/s3-storage-provider/validate_config.yml" - when: matrix_synapse_ext_synapse_s3_storage_provider_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/s3-storage-provider/setup_install.yml" - tags: - - setup-all - - setup-synapse diff --git a/roles/custom/matrix-synapse/tasks/ext/setup_uninstall.yml b/roles/custom/matrix-synapse/tasks/ext/setup_uninstall.yml index cca6c1b8..f584f584 100644 --- a/roles/custom/matrix-synapse/tasks/ext/setup_uninstall.yml +++ b/roles/custom/matrix-synapse/tasks/ext/setup_uninstall.yml @@ -1,52 +1,52 @@ --- # encryption-disabler -- block: - - when: not matrix_synapse_ext_encryption_disabler_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/encryption-disabler/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: not matrix_synapse_ext_encryption_disabler_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/encryption-disabler/setup_uninstall.yml" # rest-auth -- block: - - when: not matrix_synapse_ext_password_provider_rest_auth_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/rest-auth/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: not matrix_synapse_ext_password_provider_rest_auth_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/rest-auth/setup_uninstall.yml" # shared-secret-auth -- block: - - when: not matrix_synapse_ext_password_provider_shared_secret_auth_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/shared-secret-auth/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: not matrix_synapse_ext_password_provider_shared_secret_auth_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/shared-secret-auth/setup_uninstall.yml" # ldap-auth has no uninstall tasks # synapse-simple-antispam -- block: - - when: not matrix_synapse_ext_spam_checker_synapse_simple_antispam_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/synapse-simple-antispam/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: not matrix_synapse_ext_spam_checker_synapse_simple_antispam_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/synapse-simple-antispam/setup_uninstall.yml" # mjolnir-antispam -- block: - - when: not matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/mjolnir-antispam/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: not matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/mjolnir-antispam/setup_uninstall.yml" # s3-storage-provider -- block: - - when: not matrix_synapse_ext_synapse_s3_storage_provider_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/s3-storage-provider/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - when: not matrix_synapse_ext_synapse_s3_storage_provider_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/s3-storage-provider/setup_uninstall.yml" diff --git a/roles/custom/matrix-synapse/tasks/main.yml b/roles/custom/matrix-synapse/tasks/main.yml index d00c1808..88a54afc 100644 --- a/roles/custom/matrix-synapse/tasks/main.yml +++ b/roles/custom/matrix-synapse/tasks/main.yml @@ -1,9 +1,6 @@ --- -- block: - - when: matrix_synapse_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/init.yml" - tags: +- tags: - setup-all - setup-synapse - install-all @@ -19,55 +16,58 @@ - stop - stop-all - stop-group + block: + - when: matrix_synapse_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/init.yml" -- block: +- tags: + - setup-all + - setup-synapse + - install-all + - install-synapse + block: - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-synapse - - install-all - - install-synapse - -- block: + block: - when: not matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-synapse -- block: +- tags: + - import-synapse-media-store + block: - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/import_media_store.yml" - tags: - - import-synapse-media-store -- block: +- tags: + - register-user + block: - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/register_user.yml" - tags: - - register-user -- block: +- tags: + - update-user-password + block: - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/update_user_password.yml" - tags: - - update-user-password -- block: +- tags: + - rust-synapse-compress-state + block: - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/rust-synapse-compress-state/main.yml" - tags: - - rust-synapse-compress-state -- block: +- tags: + - self-check + block: - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_client_api.yml" - when: matrix_synapse_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/self_check_federation_api.yml" - tags: - - self-check diff --git a/roles/custom/matrix-synapse/tasks/setup_install.yml b/roles/custom/matrix-synapse/tasks/setup_install.yml index c8b191ec..378a0dee 100644 --- a/roles/custom/matrix-synapse/tasks/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/setup_install.yml @@ -23,33 +23,33 @@ # trying to chown/chmod it here will cause trouble. when: item.when | bool -- block: +- tags: + - setup-all + - setup-synapse + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/setup_install.yml" - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/setup_uninstall.yml" - tags: + +- tags: - setup-all - setup-synapse - -- block: + block: - when: matrix_synapse_workers_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/setup_install.yml" - tags: + +- tags: - setup-all - setup-synapse - -- block: + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/setup_install.yml" - tags: + +- tags: - setup-all - setup-synapse - -- block: + block: - when: matrix_s3_media_store_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/goofys/setup_install.yml" - when: not matrix_s3_media_store_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/goofys/setup_uninstall.yml" - tags: - - setup-all - - setup-synapse diff --git a/roles/custom/matrix-synapse/tasks/setup_uninstall.yml b/roles/custom/matrix-synapse/tasks/setup_uninstall.yml index 72b64c87..7ce5e13d 100644 --- a/roles/custom/matrix-synapse/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-synapse/tasks/setup_uninstall.yml @@ -1,27 +1,27 @@ --- -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/setup_uninstall.yml" - tags: +- tags: - setup-all - setup-synapse + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/ext/setup_uninstall.yml" -- block: +- tags: + - setup-all + - setup-synapse + block: - when: not matrix_synapse_workers_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/setup_uninstall.yml" - tags: + +- tags: - setup-all - setup-synapse - -- block: + block: - ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/setup_uninstall.yml" - tags: + +- tags: - setup-all - setup-synapse - -- block: + block: - when: not matrix_s3_media_store_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/goofys/setup_uninstall.yml" - tags: - - setup-all - - setup-synapse diff --git a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml index 4a62d88d..169be60b 100644 --- a/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml +++ b/roles/custom/matrix-synapse/tasks/synapse/setup_install.yml @@ -48,6 +48,7 @@ . environment: DOCKER_BUILDKIT: 1 + changed_when: true when: "matrix_synapse_git_pull_results.changed | bool or matrix_synapse_docker_image_check_result.stdout == ''" - name: Ensure Synapse Docker image is pulled @@ -82,11 +83,6 @@ path: "{{ matrix_synapse_customized_docker_src_files_path }}" nocache: "{{ matrix_synapse_docker_image_customized_build_nocache }}" -- name: Check if a Synapse signing key exists - ansible.builtin.stat: - path: "{{ matrix_synapse_config_dir_path }}/{{ matrix_server_fqn_matrix }}.signing.key" - register: matrix_synapse_signing_key_stat - # We do this so that the signing key would get generated. # # This will also generate a default homeserver.yaml configuration file and a log configuration file. @@ -95,19 +91,20 @@ # We don't use the `docker_container` module, because using it with `cap_drop` requires # a very recent docker-py version, which is not available for a lot of people yet. - name: Generate initial Synapse config and signing key - ansible.builtin.command: | - docker run - --rm - --name=matrix-config - --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} - --cap-drop=ALL - --mount type=bind,src={{ matrix_synapse_config_dir_path }},dst=/data - -e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml - -e SYNAPSE_SERVER_NAME={{ matrix_server_fqn_matrix }} - -e SYNAPSE_REPORT_STATS=no - {{ matrix_synapse_docker_image }} - generate - when: "not matrix_synapse_signing_key_stat.stat.exists" + ansible.builtin.command: + cmd: | + docker run + --rm + --name=matrix-config + --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }} + --cap-drop=ALL + --mount type=bind,src={{ matrix_synapse_config_dir_path }},dst=/data + -e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml + -e SYNAPSE_SERVER_NAME={{ matrix_server_fqn_matrix }} + -e SYNAPSE_REPORT_STATS=no + {{ matrix_synapse_docker_image }} + generate + creates: "{{ matrix_synapse_config_dir_path }}/{{ matrix_server_fqn_matrix }}.signing.key" - name: Ensure Synapse homeserver config installed ansible.builtin.copy: diff --git a/roles/custom/matrix-user-creator/tasks/main.yml b/roles/custom/matrix-user-creator/tasks/main.yml index 2198b3eb..ff93a463 100644 --- a/roles/custom/matrix-user-creator/tasks/main.yml +++ b/roles/custom/matrix-user-creator/tasks/main.yml @@ -1,10 +1,10 @@ --- -- block: - - when: matrix_user_creator_users | length > 0 - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup.yml" - tags: +- tags: # This role intentionally doesn't do work on a `setup-all` tag. # If it did, the initial installation (`--tags=setup-all`) would also potentially polute the database with data, # which would make importing a database dump problematic. - ensure-matrix-users-created + block: + - when: matrix_user_creator_users | length > 0 + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup.yml" diff --git a/roles/custom/matrix-user-verification-service/tasks/main.yml b/roles/custom/matrix-user-verification-service/tasks/main.yml index 6eaaaf23..1b65f86a 100644 --- a/roles/custom/matrix-user-verification-service/tasks/main.yml +++ b/roles/custom/matrix-user-verification-service/tasks/main.yml @@ -1,19 +1,19 @@ --- -- block: +- tags: + - setup-all + - setup-user-verification-service + - install-all + - install-user-verification-service + block: - when: run_setup | bool and matrix_user_verification_service_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: run_setup | bool and matrix_user_verification_service_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: + +- tags: - setup-all - setup-user-verification-service - - install-all - - install-user-verification-service - -- block: + block: - when: run_setup | bool and not matrix_user_verification_service_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-user-verification-service diff --git a/roles/custom/matrix_playbook_migration/tasks/main.yml b/roles/custom/matrix_playbook_migration/tasks/main.yml index 4a39e2c7..4dbd3554 100644 --- a/roles/custom/matrix_playbook_migration/tasks/main.yml +++ b/roles/custom/matrix_playbook_migration/tasks/main.yml @@ -1,22 +1,22 @@ --- -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - tags: +- tags: - setup-all - install-all + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" -- block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/cleanup_usr_local_bin.yml" - tags: +- tags: - setup-all - install-all + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/cleanup_usr_local_bin.yml" - when: devture_traefik_enabled | bool - block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/devture_traefik_to_matrix_traefik.yml" tags: - setup-all - install-all - setup-traefik - install-traefik + block: + - ansible.builtin.include_tasks: "{{ role_path }}/tasks/devture_traefik_to_matrix_traefik.yml" From cdb1452d170d99170af77687380c059a581b4570 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 7 Mar 2023 20:38:36 +0000 Subject: [PATCH 307/382] Update jitsi stable-8252 -> stable-8319 --- roles/custom/matrix-jitsi/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index cc24d356..ef4cf100 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -91,7 +91,7 @@ matrix_jitsi_jibri_recorder_password: '' matrix_jitsi_enable_lobby: false -matrix_jitsi_version: stable-8252 +matrix_jitsi_version: stable-8319 matrix_jitsi_container_image_tag: "{{ matrix_jitsi_version }}" # for backward-compatibility matrix_jitsi_web_docker_image: "{{ matrix_container_global_registry_prefix }}jitsi/web:{{ matrix_jitsi_container_image_tag }}" From 06a390b6b8b7f199e246b298e4b0c53db265f272 Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Wed, 8 Mar 2023 19:22:31 -0500 Subject: [PATCH 308/382] Update configuring-playbook-cactus-comments.md --- docs/configuring-playbook-cactus-comments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-cactus-comments.md b/docs/configuring-playbook-cactus-comments.md index 00c76f54..4d75c02e 100644 --- a/docs/configuring-playbook-cactus-comments.md +++ b/docs/configuring-playbook-cactus-comments.md @@ -24,7 +24,7 @@ matrix_cactus_comments_enabled: true # To do this you need to uncomment one of the following lines (depending if you are using synapse or dentrite as a homeserver) # If you don't know which one you use: The default is synapse ;) # matrix_synapse_allow_guest_access: true -# matrix_dentrite_allow_guest_access +# matrix_dentrite_allow_guest_access: true ``` ## Installing From f2ebb5a91d9b008b589ecf57acd2823f38d9b35f Mon Sep 17 00:00:00 2001 From: Joe Kappus Date: Thu, 9 Mar 2023 00:02:54 -0500 Subject: [PATCH 309/382] Bump ChatGPT version and add new config settings OpenAI changed the model a month ago and broke the bot. This bump fixes it and adds the default model. --- roles/custom/matrix-bot-chatgpt/defaults/main.yml | 3 ++- roles/custom/matrix-bot-chatgpt/templates/env.j2 | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index 489b22c4..13b25a00 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -9,7 +9,7 @@ matrix_bot_chatgpt_docker_repo: "https://github.com/matrixgpt/matrix-chatgpt-bot matrix_bot_chatgpt_docker_repo_version: "{{ 'latest' if matrix_bot_chatgpt_version == 'latest' else matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_src_files_path: "{{ matrix_base_data_path }}/chatgpt/docker-src" -matrix_bot_chatgpt_version: 2.2.1 +matrix_bot_chatgpt_version: 3.0.0 matrix_bot_chatgpt_docker_image: "{{ matrix_bot_chatgpt_docker_image_name_prefix }}matrixgpt/matrix-chatgpt-bot:{{ matrix_bot_chatgpt_version }}" matrix_bot_chatgpt_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_chatgpt_container_image_self_build else 'ghcr.io/' }}" matrix_bot_chatgpt_docker_image_force_pull: "{{ matrix_bot_chatgpt_docker_image.endswith(':latest') }}" @@ -29,6 +29,7 @@ matrix_bot_chatgpt_systemd_wanted_services_list: [] # ChatGPT Settings matrix_bot_chatgpt_openai_api_key: '' +matrix_bot_chatgpt_api_model: 'gpt-3.5-turbo' matrix_bot_chatgpt_context: 'thread' # CHATGPT_CONTEXT="thread" matrix_bot_chatgpt_keyv_backend: 'file' diff --git a/roles/custom/matrix-bot-chatgpt/templates/env.j2 b/roles/custom/matrix-bot-chatgpt/templates/env.j2 index 40b2f9ff..bc8c3866 100644 --- a/roles/custom/matrix-bot-chatgpt/templates/env.j2 +++ b/roles/custom/matrix-bot-chatgpt/templates/env.j2 @@ -3,6 +3,7 @@ MATRIX_ACCESS_TOKEN={{ matrix_bot_chatgpt_matrix_access_token }} OPENAI_API_KEY={{ matrix_bot_chatgpt_openai_api_key }} CHATGPT_CONTEXT={{ matrix_bot_chatgpt_context }} +CHATGPT_API_MODEL={{ matrix_bot_chatgpt_api_model }} KEYV_BACKEND={{ matrix_bot_chatgpt_keyv_backend }} KEYV_URL={{ matrix_bot_chatgpt_keyv_url }} From 70af9a44816faff432ecd913cf72cdded6c18f86 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 9 Mar 2023 08:56:51 +0200 Subject: [PATCH 310/382] Upgrade Traefik (v2.9.8-1 -> v2.9.8-2) - not using unprivileged ports anymore --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 013a6c53..05c5aa70 100644 --- a/requirements.yml +++ b/requirements.yml @@ -55,7 +55,7 @@ version: v0.1.1-1 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: v2.9.8-1 + version: v2.9.8-2 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: v2.8.1-0 From 122a834a7f271c302fc1e5ac28e51baa7eb3b733 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 10 Mar 2023 20:23:13 +0000 Subject: [PATCH 311/382] update prometheus-node-exporter 1.5.0-2 -> 1.5.0-3 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 05c5aa70..e841cb4e 100644 --- a/requirements.yml +++ b/requirements.yml @@ -31,7 +31,7 @@ version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git - version: v1.5.0-2 + version: v1.5.0-3 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.11.1-2 From bf3677a9b5a47fa1433bea1ce1e6c4ca2174c67f Mon Sep 17 00:00:00 2001 From: Aine Date: Fri, 10 Mar 2023 23:48:43 +0200 Subject: [PATCH 312/382] add synapse_auto_compressor --- README.md | 1 + ...guring-playbook-synapse-auto-compressor.md | 36 +++++++++++++ playbooks/matrix.yml | 1 + .../defaults/main.yml | 51 ++++++++++++++++++ .../tasks/install.yml | 53 +++++++++++++++++++ .../tasks/main.yml | 17 ++++++ .../tasks/uninstall.yml | 29 ++++++++++ .../matrix-synapse-auto-compressor.service.j2 | 35 ++++++++++++ .../matrix-synapse-auto-compressor.timer.j2 | 9 ++++ 9 files changed, 232 insertions(+) create mode 100644 docs/configuring-playbook-synapse-auto-compressor.md create mode 100644 roles/custom/matrix-synapse-auto-compressor/defaults/main.yml create mode 100644 roles/custom/matrix-synapse-auto-compressor/tasks/install.yml create mode 100644 roles/custom/matrix-synapse-auto-compressor/tasks/main.yml create mode 100644 roles/custom/matrix-synapse-auto-compressor/tasks/uninstall.yml create mode 100644 roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 create mode 100644 roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.timer.j2 diff --git a/README.md b/README.md index 185c4085..ff93cd1f 100644 --- a/README.md +++ b/README.md @@ -163,6 +163,7 @@ Various services that don't fit any other category. | Name | Default? | Description | Documentation | | ---- | -------- | ----------- | ------------- | | [sliding-sync](https://github.com/matrix-org/sliding-sync)| x | Sliding Sync support for clients which require it (e.g. Element X) | [Link](docs/configuring-playbook-sliding-sync-proxy.md) | +| [synapse_auto_compressor](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) | x | A cli tool that automatically compresses `state_groups` database table in background. | [Link](docs/configuring-playbook-synapse-auto-compressor.md) | | [synapse-simple-antispam](https://github.com/t2bot/synapse-simple-antispam) (advanced) | x | A spam checker module | [Link](docs/configuring-playbook-synapse-simple-antispam.md) | | [Matrix Corporal](https://github.com/devture/matrix-corporal) (advanced) | x | Reconciliator and gateway for a managed Matrix server | [Link](docs/configuring-playbook-matrix-corporal.md) | | [Etherpad](https://etherpad.org) | x | An open source collaborative text editor | [Link](docs/configuring-playbook-etherpad.md) | diff --git a/docs/configuring-playbook-synapse-auto-compressor.md b/docs/configuring-playbook-synapse-auto-compressor.md new file mode 100644 index 00000000..cd6079ca --- /dev/null +++ b/docs/configuring-playbook-synapse-auto-compressor.md @@ -0,0 +1,36 @@ +# Setting up synapse_auto_compressor + +The playbook can install and configure [synapse_auto_compressor](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) for you. + +It's a cli tool that automatically compresses `state_groups` database table in background. + +See the project's [documentation](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) to learn what it does and why it might be useful to you. + + +## Adjusting the playbook configuration + +Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file: + +```yaml +matrix_synapse_auto_compressor_enabled: true +``` + + +## Installing + +After configuring the playbook, run the [installation](installing.md) command again: + +``` +ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start +``` + + +## Usage + +After installation, synapse_auto_compressor will run automatically every day by the defined schedule + +## Manually start the tool + +For testing your setup it can be helpful to not wait until 00:00. If you want to run the tool immediately, log onto the server +and run `systemctl start matrix-synapse-auto-compressor`. This will not return until the run is done, so possibly a long time. +Consider using [tmux](https://en.wikipedia.org/wiki/Tmux) if your SSH connection is unstable. diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index a2b7f2b8..b8964aa5 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -84,6 +84,7 @@ - custom/matrix-cactus-comments - custom/matrix-rageshake - custom/matrix-synapse + - custom/matrix-synapse-auto-compressor - custom/matrix-synapse-reverse-proxy-companion - custom/matrix-dendrite - custom/matrix-conduit diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml new file mode 100644 index 00000000..d24564ce --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -0,0 +1,51 @@ +--- +# synapse_auto_compressor tool +# Project source code URL: https://github.com/matrix-org/rust-synapse-compress-state + +matrix_synapse_auto_compressor_enabled: false + +matrix_synapse_auto_compressor_container_image_self_build: false +matrix_synapse_auto_compressor_container_repo: "https://gitlab.com/etke.cc/rust-synapse-compress-state.git" +matrix_synapse_auto_compressor_container_repo_version: "{{ 'main' if matrix_synapse_auto_compressor_version == 'latest' else matrix_synapse_auto_compressor_version }}" +matrix_synapse_auto_compressor_container_src_files_path: "{{ matrix_synapse_auto_compressor_base_path }}" + +matrix_synapse_auto_compressor_version: v0.1.3 +matrix_synapse_auto_compressor_container_image: "{{ matrix_synapse_auto_compressor_container_image_name_prefix }}etke.cc/rust-synapse-compress-state:{{ matrix_synapse_auto_compressor_version }}" +matrix_synapse_auto_compressor_container_image_name_prefix: "{{ 'localhost/' if matrix_synapse_auto_compressor_container_image_self_build else 'registry.gitlab.com/' }}" +matrix_synapse_auto_compressor_container_image_force_pull: "{{ matrix_synapse_auto_compressor_container_image.endswith(':latest') }}" + +matrix_synapse_auto_compressor_base_path: "{{ matrix_base_data_path }}/synapse-auto-compressor" + +# A list of extra arguments to pass to the container +matrix_synapse_auto_compressor_container_extra_arguments: [] + +# List of systemd services that matrix-synapse-auto-compressor.service depends on +matrix_synapse_auto_compressor_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-synapse-auto-compressor.service wants +matrix_synapse_auto_compressor_systemd_wanted_services_list: [] + +matrix_synapse_auto_compressor_database_username: 'synapse' +matrix_synapse_auto_compressor_database_password: 'some-password' +matrix_synapse_auto_compressor_database_hostname: '' +matrix_synapse_auto_compressor_database_port: 5432 +matrix_synapse_auto_compressor_database_name: 'synapse' + +# connection string to synapse database (postgres only) +matrix_synapse_auto_compressor_synapse_database: 'postgres://{{ matrix_synapse_auto_compressor_database_username }}:{{ matrix_synapse_auto_compressor_database_password }}@{{ matrix_synapse_auto_compressor_database_hostname }}:{{ matrix_synapse_auto_compressor_database_port }}/{{ matrix_synapse_auto_compressor_database_name }}' + +# systemd calendar configuration for the compressor job +matrix_synapse_auto_compressor_calendar: "*-*-* 00:00:00" + +# The number of state groups to work on at once. +# All of the entries from state_groups_state are requested from the database for state groups that are worked on. +# Therefore small chunk sizes may be needed on machines with low memory. +# Note: if the compressor fails to find space savings on the chunk as a whole +# (which may well happen in rooms with lots of backfill in) then the entire chunk is skipped. +matrix_synapse_auto_compressor_chunk_size: 500 + +# CHUNKS_TO_COMPRESS chunks of size CHUNK_SIZE will be compressed. +# The higher this number is set to, the longer the compressor will run for. +matrix_synapse_auto_compressor_chunks_to_compress: 100 + +matrix_synapse_auto_compressor_command: "synapse_auto_compressor -p {{ matrix_synapse_auto_compressor_synapse_database }} -c {{ matrix_synapse_auto_compressor_chunk_size }} -n {{ matrix_synapse_auto_compressor_chunks_to_compress }}" diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml new file mode 100644 index 00000000..ec825b13 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml @@ -0,0 +1,53 @@ +--- +- name: Ensure synapse-auto-compressor paths exist + ansible.builtin.file: + path: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: matrix_synapse_auto_compressor_container_image_self_build | bool + +- name: Ensure synapse-auto-compressor image is pulled + community.docker.docker_image: + name: "{{ matrix_synapse_auto_compressor_container_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_synapse_auto_compressor_container_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_synapse_auto_compressor_container_image_force_pull }}" + when: "not matrix_synapse_auto_compressor_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure synapse-auto-compressor repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_synapse_auto_compressor_container_repo }}" + version: "{{ matrix_synapse_auto_compressor_container_repo_version }}" + dest: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_synapse_auto_compressor_git_pull_results + when: "matrix_synapse_auto_compressor_container_image_self_build | bool" + +- name: Ensure synapse-auto-compressor image is built + community.docker.docker_image: + name: "{{ matrix_synapse_auto_compressor_container_image }}" + source: build + force_source: "{{ matrix_synapse_auto_compressor_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + pull: true + when: "matrix_synapse_auto_compressor_container_image_self_build | bool" + +- name: Ensure matrix-synapse-auto-compressor systemd service and timer are installed + ansible.builtin.template: + src: "{{ role_path }}/templates/matrix-synapse-auto-compressor.{{ item }}.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.{{ item }}" + mode: 0644 + with_items: + - service + - timer diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml new file mode 100644 index 00000000..b3549bd6 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml @@ -0,0 +1,17 @@ +--- + +- block: + - when: matrix_synapse_auto_compressor_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" + tags: + - setup-all + - setup-synapse-auto-compressor + - install-all + - install-synapse-auto-compressor + +- block: + - when: not matrix_synapse_auto_compressor_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" + tags: + - setup-all + - setup-synapse-auto-compressor diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/uninstall.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/uninstall.yml new file mode 100644 index 00000000..075f3c10 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/uninstall.yml @@ -0,0 +1,29 @@ +--- +- name: Check existence of matrix-synapse-auto-compressor service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.service" + register: matrix_synapse_auto_compressor_service_stat + +- when: matrix_synapse_auto_compressor_service_stat.stat.exists | bool + block: + - name: Ensure matrix-synapse-auto-compressor is stopped + ansible.builtin.service: + name: matrix-synapse-auto-compressor + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-synapse-auto-compressor.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.service" + state: absent + + - name: Ensure matrix-synapse-auto-compressor.timer doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.timer" + state: absent + + - name: Ensure Matrix synapse-auto-compressor paths don't exist + ansible.builtin.file: + path: "{{ matrix_synapse_auto_compressor_base_path }}" + state: absent diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 new file mode 100644 index 00000000..02aef84b --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 @@ -0,0 +1,35 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Synapse State Auto Compressor +{% for service in matrix_synapse_auto_compressor_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_synapse_auto_compressor_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=oneshot +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-auto-compressor 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-auto-compressor 2>/dev/null || true' +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-synapse-auto-compressor \ + --log-driver=none \ + --cap-drop=ALL \ + --read-only \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --network={{ matrix_docker_network }} \ + {% for arg in matrix_synapse_auto_compressor_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_synapse_auto_compressor_container_image }} \ + {{ matrix_synapse_auto_compressor_command }} + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-auto-compressor 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-auto-compressor 2>/dev/null || true' +SyslogIdentifier=matrix-synapse-auto-compressor + +[Install] +WantedBy=multi-user.target diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.timer.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.timer.j2 new file mode 100644 index 00000000..1d7b16d3 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.timer.j2 @@ -0,0 +1,9 @@ +[Unit] +Description=Synapse State Auto Compressor Timer + +[Timer] +Unit=matrix-synapse-auto-compressor.service +OnCalendar={{ matrix_synapse_auto_compressor_calendar }} + +[Install] +WantedBy=timers.target From 43ae0b2943e24444369c9f6f534f5890238c9cf7 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 09:44:43 +0200 Subject: [PATCH 313/382] Minor rewording --- docs/configuring-playbook-synapse-auto-compressor.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/configuring-playbook-synapse-auto-compressor.md b/docs/configuring-playbook-synapse-auto-compressor.md index cd6079ca..108b6a11 100644 --- a/docs/configuring-playbook-synapse-auto-compressor.md +++ b/docs/configuring-playbook-synapse-auto-compressor.md @@ -2,7 +2,7 @@ The playbook can install and configure [synapse_auto_compressor](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) for you. -It's a cli tool that automatically compresses `state_groups` database table in background. +It's a CLI tool that automatically compresses Synapse's `state_groups` database table in the background. See the project's [documentation](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) to learn what it does and why it might be useful to you. @@ -27,10 +27,10 @@ ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start ## Usage -After installation, synapse_auto_compressor will run automatically every day by the defined schedule +After installation, `synapse_auto_compressor` will run automatically every day at `00:00:00` (as defined in `matrix_synapse_auto_compressor_calendar` by default). ## Manually start the tool For testing your setup it can be helpful to not wait until 00:00. If you want to run the tool immediately, log onto the server -and run `systemctl start matrix-synapse-auto-compressor`. This will not return until the run is done, so possibly a long time. +and run `systemctl start matrix-synapse-auto-compressor`. Running this command will not return control to your terminal until the compression run is done, which may take a long time. Consider using [tmux](https://en.wikipedia.org/wiki/Tmux) if your SSH connection is unstable. From b28d779c6cc8cc4f80a48295ebd0a93cc63311c5 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 09:48:46 +0200 Subject: [PATCH 314/382] Add matrix-synapse-auto-compressor section in group_vars/matrix_servers --- group_vars/matrix_servers | 18 ++++++++++++++++++ .../defaults/main.yml | 3 ++- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 85fbfd2a..6a90114f 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3326,6 +3326,24 @@ matrix_synapse_app_service_config_files_auto: "{{ matrix_homeserver_app_service_ # ###################################################################### + +###################################################################### +# +# matrix-synapse-auto-compressor +# +###################################################################### + +matrix_synapse_auto_compressor_enabled: false + +matrix_synapse_auto_compressor_container_image_self_build: "{{ matrix_architecture not in ['arm32', 'arm64', 'amd64'] }}" + +###################################################################### +# +# /matrix-synapse-auto-compressor +# +###################################################################### + + ###################################################################### # # matrix-synapse-reverse-proxy-companion diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml index d24564ce..99fb272f 100644 --- a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -1,8 +1,9 @@ --- + # synapse_auto_compressor tool # Project source code URL: https://github.com/matrix-org/rust-synapse-compress-state -matrix_synapse_auto_compressor_enabled: false +matrix_synapse_auto_compressor_enabled: true matrix_synapse_auto_compressor_container_image_self_build: false matrix_synapse_auto_compressor_container_repo: "https://gitlab.com/etke.cc/rust-synapse-compress-state.git" From ca69fce64824548d240f69b375f1d6f51cba41b1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 09:55:32 +0200 Subject: [PATCH 315/382] Add missing group vars for matrix-synapse-auto-compressor --- group_vars/matrix_servers | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6a90114f..5712b037 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3333,9 +3333,23 @@ matrix_synapse_app_service_config_files_auto: "{{ matrix_homeserver_app_service_ # ###################################################################### +# Not enabled by default, for now matrix_synapse_auto_compressor_enabled: false -matrix_synapse_auto_compressor_container_image_self_build: "{{ matrix_architecture not in ['arm32', 'arm64', 'amd64'] }}" +matrix_synapse_auto_compressor_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" + +matrix_synapse_auto_compressor_database_username: "{{ matrix_synapse_database_user if matrix_synapse_enabled else '' }}" +matrix_synapse_auto_compressor_database_password: "{{ matrix_synapse_database_password if matrix_synapse_enabled else '' }}" +matrix_synapse_auto_compressor_database_hostname: "{{ matrix_synapse_database_host if matrix_synapse_enabled else '' }}" +matrix_synapse_auto_compressor_database_port: "{{ matrix_synapse_database_port if matrix_synapse_enabled else '5432' }}" +matrix_synapse_auto_compressor_database_name: "{{ matrix_synapse_database_database if matrix_synapse_enabled else '' }}" + +matrix_synapse_auto_compressor_systemd_required_services_list: | + {{ + ['docker.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + }} ###################################################################### # From 0f2e568b3098e63ca08af94416d0da37ce4a56ac Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 09:56:53 +0200 Subject: [PATCH 316/382] Add validation tasks to matrix-synapse-auto-compressor --- .../matrix-synapse-auto-compressor/defaults/main.yml | 2 +- .../matrix-synapse-auto-compressor/tasks/main.yml | 3 +++ .../tasks/validate_config.yml | 10 ++++++++++ 3 files changed, 14 insertions(+), 1 deletion(-) create mode 100644 roles/custom/matrix-synapse-auto-compressor/tasks/validate_config.yml diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml index 99fb272f..8de4097e 100644 --- a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -27,7 +27,7 @@ matrix_synapse_auto_compressor_systemd_required_services_list: ['docker.service' matrix_synapse_auto_compressor_systemd_wanted_services_list: [] matrix_synapse_auto_compressor_database_username: 'synapse' -matrix_synapse_auto_compressor_database_password: 'some-password' +matrix_synapse_auto_compressor_database_password: '' matrix_synapse_auto_compressor_database_hostname: '' matrix_synapse_auto_compressor_database_port: 5432 matrix_synapse_auto_compressor_database_name: 'synapse' diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml index b3549bd6..2cce1f12 100644 --- a/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml @@ -1,6 +1,9 @@ --- - block: + - when: matrix_synapse_auto_compressor_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + - when: matrix_synapse_auto_compressor_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" tags: diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/validate_config.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/validate_config.yml new file mode 100644 index 00000000..b0dbfab0 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/validate_config.yml @@ -0,0 +1,10 @@ +--- + +- name: Fail if required matrix-synapse-auto-compressor settings not defined + ansible.builtin.fail: + msg: > + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_synapse_auto_compressor_database_hostname + - matrix_synapse_auto_compressor_database_password From 26d5719df414d26f3af8470cac5a36d1f3e65eed Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 10:01:51 +0200 Subject: [PATCH 317/382] Make matrix-synapse-auto-compressor live in its own container network It will, additionally, be connected to the devture-postgres network, if devture-postgres is enabled. --- group_vars/matrix_servers | 5 +++++ .../defaults/main.yml | 8 ++++++++ .../tasks/install.yml | 5 +++++ .../matrix-synapse-auto-compressor.service.j2 | 13 +++++++++++-- 4 files changed, 29 insertions(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 5712b037..c96b34a7 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3338,6 +3338,11 @@ matrix_synapse_auto_compressor_enabled: false matrix_synapse_auto_compressor_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" +matrix_synapse_auto_compressor_container_additional_networks: | + {{ + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != matrix_synapse_auto_compressor_container_network else []) + }} + matrix_synapse_auto_compressor_database_username: "{{ matrix_synapse_database_user if matrix_synapse_enabled else '' }}" matrix_synapse_auto_compressor_database_password: "{{ matrix_synapse_database_password if matrix_synapse_enabled else '' }}" matrix_synapse_auto_compressor_database_hostname: "{{ matrix_synapse_database_host if matrix_synapse_enabled else '' }}" diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml index 8de4097e..0649392d 100644 --- a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -17,6 +17,14 @@ matrix_synapse_auto_compressor_container_image_force_pull: "{{ matrix_synapse_au matrix_synapse_auto_compressor_base_path: "{{ matrix_base_data_path }}/synapse-auto-compressor" +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_synapse_auto_compressor_container_network: matrix-synapse-auto-compressor + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_synapse_auto_compressor_container_additional_networks: [] + # A list of extra arguments to pass to the container matrix_synapse_auto_compressor_container_extra_arguments: [] diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml index ec825b13..6f4524bb 100644 --- a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml @@ -43,6 +43,11 @@ pull: true when: "matrix_synapse_auto_compressor_container_image_self_build | bool" +- name: Ensure matrix-synapse-auto-compressor container network is created + community.general.docker_network: + name: "{{ matrix_synapse_auto_compressor_container_network }}" + driver: bridge + - name: Ensure matrix-synapse-auto-compressor systemd service and timer are installed ansible.builtin.template: src: "{{ role_path }}/templates/matrix-synapse-auto-compressor.{{ item }}.j2" diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 index 02aef84b..e769438d 100644 --- a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 +++ b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 @@ -15,18 +15,27 @@ Type=oneshot Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-auto-compressor 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-auto-compressor 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-synapse-auto-compressor \ + +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-synapse-auto-compressor \ --log-driver=none \ --cap-drop=ALL \ --read-only \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_synapse_auto_compressor_container_network }} \ {% for arg in matrix_synapse_auto_compressor_container_extra_arguments %} {{ arg }} \ {% endfor %} {{ matrix_synapse_auto_compressor_container_image }} \ {{ matrix_synapse_auto_compressor_command }} +{% for network in matrix_synapse_auto_compressor_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-synapse-auto-compressor +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-synapse-auto-compressor + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-auto-compressor 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-auto-compressor 2>/dev/null || true' SyslogIdentifier=matrix-synapse-auto-compressor From 328d0d8a5f47c4f4e15d70cb4c796dbd6a957b6b Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 10:17:42 +0200 Subject: [PATCH 318/382] Move synapse-auto-compressor Postgres argument to an environment variable This provides an additional security benefit. The password won't leak in the process list anymore. --- .../defaults/main.yml | 14 +++++++++----- .../tasks/install.yml | 18 ++++++++++++++++-- .../templates/env.j2 | 1 + .../matrix-synapse-auto-compressor.service.j2 | 4 +++- 4 files changed, 29 insertions(+), 8 deletions(-) create mode 100644 roles/custom/matrix-synapse-auto-compressor/templates/env.j2 diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml index 0649392d..61194714 100644 --- a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -5,18 +5,19 @@ matrix_synapse_auto_compressor_enabled: true +matrix_synapse_auto_compressor_version: v0.1.3 + +matrix_synapse_auto_compressor_base_path: "{{ matrix_base_data_path }}/synapse-auto-compressor" +matrix_synapse_auto_compressor_container_src_files_path: "{{ matrix_synapse_auto_compressor_base_path }}/container-src" + matrix_synapse_auto_compressor_container_image_self_build: false matrix_synapse_auto_compressor_container_repo: "https://gitlab.com/etke.cc/rust-synapse-compress-state.git" matrix_synapse_auto_compressor_container_repo_version: "{{ 'main' if matrix_synapse_auto_compressor_version == 'latest' else matrix_synapse_auto_compressor_version }}" -matrix_synapse_auto_compressor_container_src_files_path: "{{ matrix_synapse_auto_compressor_base_path }}" -matrix_synapse_auto_compressor_version: v0.1.3 matrix_synapse_auto_compressor_container_image: "{{ matrix_synapse_auto_compressor_container_image_name_prefix }}etke.cc/rust-synapse-compress-state:{{ matrix_synapse_auto_compressor_version }}" matrix_synapse_auto_compressor_container_image_name_prefix: "{{ 'localhost/' if matrix_synapse_auto_compressor_container_image_self_build else 'registry.gitlab.com/' }}" matrix_synapse_auto_compressor_container_image_force_pull: "{{ matrix_synapse_auto_compressor_container_image.endswith(':latest') }}" -matrix_synapse_auto_compressor_base_path: "{{ matrix_base_data_path }}/synapse-auto-compressor" - # The base container network. It will be auto-created by this role if it doesn't exist already. matrix_synapse_auto_compressor_container_network: matrix-synapse-auto-compressor @@ -57,4 +58,7 @@ matrix_synapse_auto_compressor_chunk_size: 500 # The higher this number is set to, the longer the compressor will run for. matrix_synapse_auto_compressor_chunks_to_compress: 100 -matrix_synapse_auto_compressor_command: "synapse_auto_compressor -p {{ matrix_synapse_auto_compressor_synapse_database }} -c {{ matrix_synapse_auto_compressor_chunk_size }} -n {{ matrix_synapse_auto_compressor_chunks_to_compress }}" +matrix_synapse_auto_compressor_command: "synapse_auto_compressor -p $POSTGRES_LOCATION -c {{ matrix_synapse_auto_compressor_chunk_size }} -n {{ matrix_synapse_auto_compressor_chunks_to_compress }}" + +# Controls the POSTGRES_LOCATION environment variable +matrix_synapse_auto_compressor_environment_variable_postgres_location: "{{ matrix_synapse_auto_compressor_synapse_database }}" diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml index 6f4524bb..494a5678 100644 --- a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml @@ -1,12 +1,26 @@ --- + - name: Ensure synapse-auto-compressor paths exist ansible.builtin.file: - path: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + path: "{{ item.path }}" state: directory mode: 0750 owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" - when: matrix_synapse_auto_compressor_container_image_self_build | bool + when: item.when | bool + with_items: + - path: "{{ matrix_synapse_auto_compressor_base_path }}" + when: true + - path: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + when: "{{ matrix_synapse_auto_compressor_container_image_self_build }}" + +- name: Ensure synapse-auto-compressor labels installed + ansible.builtin.template: + src: "{{ role_path }}/templates/env.j2" + dest: "{{ matrix_synapse_auto_compressor_base_path }}/env" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" - name: Ensure synapse-auto-compressor image is pulled community.docker.docker_image: diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/env.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/env.j2 new file mode 100644 index 00000000..27fb1dd8 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/templates/env.j2 @@ -0,0 +1 @@ +POSTGRES_LOCATION={{ matrix_synapse_auto_compressor_environment_variable_postgres_location }} diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 index e769438d..f530d5b2 100644 --- a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 +++ b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 @@ -24,11 +24,13 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --read-only \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --network={{ matrix_synapse_auto_compressor_container_network }} \ + --env-file={{ matrix_synapse_auto_compressor_base_path }}/env \ + --entrypoint=/bin/sh \ {% for arg in matrix_synapse_auto_compressor_container_extra_arguments %} {{ arg }} \ {% endfor %} {{ matrix_synapse_auto_compressor_container_image }} \ - {{ matrix_synapse_auto_compressor_command }} + -c '{{ matrix_synapse_auto_compressor_command }}' {% for network in matrix_synapse_auto_compressor_container_additional_networks %} ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-synapse-auto-compressor From 7422337c26c464242b0295aad2c64c1735f65bf1 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 10:18:16 +0200 Subject: [PATCH 319/382] Add missing matrix-synapse-auto-compressor.timer in systemd service list --- group_vars/matrix_servers | 2 ++ 1 file changed, 2 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index c96b34a7..60c8349e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -338,6 +338,8 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-synapse-s3-storage-provider-migrate.timer', 'priority': 5000, 'groups': ['matrix']}] if matrix_synapse_ext_synapse_s3_storage_provider_enabled else []) + + ([{'name': 'matrix-synapse-auto-compressor.timer', 'priority': 5000, 'groups': ['matrix', 'synapse-auto-compressor']}] if matrix_synapse_auto_compressor_enabled else []) + + ([{'name': 'matrix-synapse-admin.service', 'priority': 4000, 'groups': ['matrix', 'synapse-admin']}] if matrix_synapse_admin_enabled else []) + ([{'name': 'matrix-synapse-reverse-proxy-companion.service', 'priority': 1500, 'groups': ['matrix', 'homeservers', 'synapse', 'reverse-proxies']}] if matrix_synapse_reverse_proxy_companion_enabled else []) From bb89d7b3fb063339c40dad937cf5a275603e868d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 10:39:19 +0200 Subject: [PATCH 320/382] Fix self-building for matrix-synapse-auto-compressor --- roles/custom/matrix-synapse-auto-compressor/defaults/main.yml | 4 ++-- roles/custom/matrix-synapse-auto-compressor/tasks/install.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml index 61194714..48ed1b03 100644 --- a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -11,8 +11,8 @@ matrix_synapse_auto_compressor_base_path: "{{ matrix_base_data_path }}/synapse-a matrix_synapse_auto_compressor_container_src_files_path: "{{ matrix_synapse_auto_compressor_base_path }}/container-src" matrix_synapse_auto_compressor_container_image_self_build: false -matrix_synapse_auto_compressor_container_repo: "https://gitlab.com/etke.cc/rust-synapse-compress-state.git" -matrix_synapse_auto_compressor_container_repo_version: "{{ 'main' if matrix_synapse_auto_compressor_version == 'latest' else matrix_synapse_auto_compressor_version }}" +matrix_synapse_auto_compressor_container_image_self_build_repo: "https://github.com/matrix-org/rust-synapse-compress-state.git" +matrix_synapse_auto_compressor_container_image_self_build_version: "{{ 'main' if matrix_synapse_auto_compressor_version == 'latest' else matrix_synapse_auto_compressor_version }}" matrix_synapse_auto_compressor_container_image: "{{ matrix_synapse_auto_compressor_container_image_name_prefix }}etke.cc/rust-synapse-compress-state:{{ matrix_synapse_auto_compressor_version }}" matrix_synapse_auto_compressor_container_image_name_prefix: "{{ 'localhost/' if matrix_synapse_auto_compressor_container_image_self_build else 'registry.gitlab.com/' }}" diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml index 494a5678..be4d5aad 100644 --- a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml @@ -36,8 +36,8 @@ - name: Ensure synapse-auto-compressor repository is present on self-build ansible.builtin.git: - repo: "{{ matrix_synapse_auto_compressor_container_repo }}" - version: "{{ matrix_synapse_auto_compressor_container_repo_version }}" + repo: "{{ matrix_synapse_auto_compressor_container_image_self_build_repo }}" + version: "{{ matrix_synapse_auto_compressor_container_image_self_build_version }}" dest: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" force: "yes" become: true From d3a0ef22b61b97d9242efc9a2021296983fd47a6 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 12 Mar 2023 10:55:08 +0200 Subject: [PATCH 321/382] Announce matrix-synapse-auto-compressor --- CHANGELOG.md | 16 ++++++++++++++++ docs/configuring-playbook.md | 2 ++ docs/maintenance-synapse.md | 4 +++- 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e9e2545..9ba17ea3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,19 @@ +# 2023-03-12 + +## synapse-auto-compressor support + +Thanks to [Aine](https://gitlab.com/etke.cc) of [etke.cc](https://etke.cc/), the playbook can now set up [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state)'s `synapse_auto_compressor` tool to run periodically. + +If enabled, `synapse_auto_compressor` runs on a schedule and compresses your Synapse database's `state_groups` table. It was possible to run `rust-synapse-compress-state` manually via the playbook even before - see [Compressing state with rust-synapse-compress-state](docs/maintenance-synapse.md#compressing-state-with-rust-synapse-compress-state). However, using `synapse_auto_compressor` is better, because: + +- it runs on a more up-to-date version of `rust-synapse-compress-state` +- it's a set-it-and-forget-it tool that you can enable and never have to deal with manual compression anymore + +This tool needs to be enabled manually, for now. In the future, we're considering enabling it by default for all Synapse installations. + +See our [Setting up synapse-auto-compressor](docs/configuring-playbook-synapse-auto-compressor.md) documentation to get started. + + # 2023-03-07 ## Sliding Sync Proxy (Element X) support diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index 68a2658a..a65858f0 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -195,6 +195,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins ### Other specialized services +- [Setting up synapse-auto-compressor](configuring-playbook-synapse-auto-compressor.md) for compressing the database on Synapse homeservers (optional) + - [Setting up the Sliding Sync Proxy](configuring-playbook-sliding-sync-proxy.md) for clients which require Sliding Sync support (like Element X) (optional) - [Setting up the Sygnal push gateway](configuring-playbook-sygnal.md) (optional) diff --git a/docs/maintenance-synapse.md b/docs/maintenance-synapse.md index a51811ff..7c3ecc1b 100644 --- a/docs/maintenance-synapse.md +++ b/docs/maintenance-synapse.md @@ -29,7 +29,9 @@ After deleting data, you may wish to run a [`FULL` Postgres `VACUUM`](./maintena [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state) can be used to optimize some `_state` tables used by Synapse. If your server participates in large rooms this is the most effective way to reduce the size of your database. -This tool should be safe to use (even when Synapse is running), but it's always a good idea to [make Postgres backups](./maintenance-postgres.md#backing-up-postgresql) first. +**Note**: besides running the `rust-synapse-compress-state` tool manually, you can also enable its `synapse-auto-compressor` tool by [Setting up synapse-auto-compressor](configuring-playbook-synapse-auto-compressor.md). The automatic tool will run on a schedule every day and you won't have to compress state manually ever again. + +`rust-synapse-compress-state` should be safe to use (even when Synapse is running), but it's always a good idea to [make Postgres backups](./maintenance-postgres.md#backing-up-postgresql) first. To ask the playbook to run rust-synapse-compress-state, execute: From cf7f9701b731640e3fde8e8d48914ecfbbf7e200 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 13 Mar 2023 11:59:14 +0200 Subject: [PATCH 322/382] urlencode matrix_synapse_auto_compressor_database_username/matrix_synapse_auto_compressor_database_password --- roles/custom/matrix-synapse-auto-compressor/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml index 48ed1b03..7b5ea54d 100644 --- a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -42,7 +42,7 @@ matrix_synapse_auto_compressor_database_port: 5432 matrix_synapse_auto_compressor_database_name: 'synapse' # connection string to synapse database (postgres only) -matrix_synapse_auto_compressor_synapse_database: 'postgres://{{ matrix_synapse_auto_compressor_database_username }}:{{ matrix_synapse_auto_compressor_database_password }}@{{ matrix_synapse_auto_compressor_database_hostname }}:{{ matrix_synapse_auto_compressor_database_port }}/{{ matrix_synapse_auto_compressor_database_name }}' +matrix_synapse_auto_compressor_synapse_database: 'postgres://{{ matrix_synapse_auto_compressor_database_username | urlencode() }}:{{ matrix_synapse_auto_compressor_database_password | urlencode() }}@{{ matrix_synapse_auto_compressor_database_hostname }}:{{ matrix_synapse_auto_compressor_database_port }}/{{ matrix_synapse_auto_compressor_database_name }}' # systemd calendar configuration for the compressor job matrix_synapse_auto_compressor_calendar: "*-*-* 00:00:00" From 40f067148d50a4b6d3c7ac0b9a399892b901085d Mon Sep 17 00:00:00 2001 From: array-in-a-matrix Date: Mon, 13 Mar 2023 15:28:58 -0400 Subject: [PATCH 323/382] added dynamic thumbnails var --- roles/custom/matrix-dendrite/defaults/main.yml | 4 ++++ .../matrix-dendrite/templates/dendrite/dendrite.yaml.j2 | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 8c2dded4..71532260 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -211,3 +211,7 @@ matrix_dendrite_userapi_auto_join_rooms: [] # statistics reporting configuration. These statistics contain the server # name, number of active users and some information on your deployment config. matrix_dendrite_report_stats: false + +# dynamically generate thumbnails for media content +matrix_dendrite_mediaapi_dynamic_thumbnails: false +matrix_dendrite_mediaapi_max_thumbnail_generators: 10 diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index 005f9bd6..e1d005b5 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -293,10 +293,10 @@ media_api: max_file_size_bytes: {{ matrix_dendrite_max_file_size_bytes|to_json }} # Whether to dynamically generate thumbnails if needed. - dynamic_thumbnails: false + dynamic_thumbnails: {{ matrix_dendrite_mediaapi_dynamic_thumbnails }} # The maximum number of simultaneous thumbnail generators to run. - max_thumbnail_generators: 10 + max_thumbnail_generators: {{ matrix_dendrite_mediaapi_max_thumbnails_generators }} # A list of thumbnail sizes to be generated for media content. thumbnail_sizes: From 734848a27bf5c3e578387e060d733520047472ce Mon Sep 17 00:00:00 2001 From: array-in-a-matrix Date: Mon, 13 Mar 2023 15:54:07 -0400 Subject: [PATCH 324/382] added full text search for dendrite --- roles/custom/matrix-dendrite/defaults/main.yml | 3 +++ .../custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 8c2dded4..8688965d 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -211,3 +211,6 @@ matrix_dendrite_userapi_auto_join_rooms: [] # statistics reporting configuration. These statistics contain the server # name, number of active users and some information on your deployment config. matrix_dendrite_report_stats: false + +# Configuration for the full-text search engine. +matrix_dendrite_syncapi_search_enabled: false diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index 005f9bd6..1960e19b 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -355,7 +355,7 @@ sync_api: # Configuration for the full-text search engine. search: # Whether or not search is enabled. - enabled: false + enabled: {{ matrix_dendrite_syncapi_search_enabled }} # The path where the search index will be created in. index_path: "/matrix-media-store-parent/searchindex" # The language most likely to be used on the server - used when indexing, to From d0aa5a1f6ac9fdafc8ef11ecd660c671f3b6f6dc Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Mon, 13 Mar 2023 16:11:16 -0400 Subject: [PATCH 325/382] Update main.yml --- roles/custom/matrix-dendrite/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 8c2dded4..4b4b08c4 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -6,7 +6,7 @@ matrix_dendrite_enabled: true matrix_dendrite_docker_image: "{{ matrix_dendrite_docker_image_name_prefix }}matrixdotorg/dendrite-monolith:{{ matrix_dendrite_docker_image_tag }}" matrix_dendrite_docker_image_name_prefix: "docker.io/" -matrix_dendrite_docker_image_tag: "v0.11.1" +matrix_dendrite_docker_image_tag: "v0.12.0" matrix_dendrite_docker_image_force_pull: "{{ matrix_dendrite_docker_image.endswith(':latest') }}" matrix_dendrite_base_path: "{{ matrix_base_data_path }}/dendrite" From c93eb1dbf7333ae55cb216a1cd3f76f8dc37824f Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Mon, 13 Mar 2023 16:19:49 -0400 Subject: [PATCH 326/382] Update dendrite.yaml.j2 --- .../matrix-dendrite/templates/dendrite/dendrite.yaml.j2 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index e1d005b5..7098a5a1 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -293,10 +293,10 @@ media_api: max_file_size_bytes: {{ matrix_dendrite_max_file_size_bytes|to_json }} # Whether to dynamically generate thumbnails if needed. - dynamic_thumbnails: {{ matrix_dendrite_mediaapi_dynamic_thumbnails }} + dynamic_thumbnails: {{ matrix_dendrite_mediaapi_dynamic_thumbnails|to_json }} # The maximum number of simultaneous thumbnail generators to run. - max_thumbnail_generators: {{ matrix_dendrite_mediaapi_max_thumbnails_generators }} + max_thumbnail_generators: {{ matrix_dendrite_mediaapi_max_thumbnails_generators|to_json }} # A list of thumbnail sizes to be generated for media content. thumbnail_sizes: From 4f0f6bd99570ad226946d02680084d31294b5ff8 Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Mon, 13 Mar 2023 16:21:18 -0400 Subject: [PATCH 327/382] Update dendrite.yaml.j2 --- .../custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index 1960e19b..8e91ae6c 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -355,7 +355,7 @@ sync_api: # Configuration for the full-text search engine. search: # Whether or not search is enabled. - enabled: {{ matrix_dendrite_syncapi_search_enabled }} + enabled: {{ matrix_dendrite_syncapi_search_enabled|to_json }} # The path where the search index will be created in. index_path: "/matrix-media-store-parent/searchindex" # The language most likely to be used on the server - used when indexing, to From fd2f497ac072f2a1bed88b2073e5f660abcbc4dc Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Mar 2023 08:47:01 +0200 Subject: [PATCH 328/382] Fix variable names Related to: - https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2565 - https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2566 --- roles/custom/matrix-dendrite/defaults/main.yml | 6 +++--- .../matrix-dendrite/templates/dendrite/dendrite.yaml.j2 | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index f95568d2..758a56e8 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -213,8 +213,8 @@ matrix_dendrite_userapi_auto_join_rooms: [] matrix_dendrite_report_stats: false # Contorls whether thumbnails for media content are generated dynamically -matrix_dendrite_mediaapi_dynamic_thumbnails: false -matrix_dendrite_mediaapi_max_thumbnail_generators: 10 +matrix_dendrite_media_api_dynamic_thumbnails: false +matrix_dendrite_media_api_max_thumbnail_generators: 10 # Controls whether the full-text search engine is enabled -matrix_dendrite_syncapi_search_enabled: false +matrix_dendrite_sync_api_search_enabled: false diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index dd92e7f5..1b31584d 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -293,10 +293,10 @@ media_api: max_file_size_bytes: {{ matrix_dendrite_max_file_size_bytes|to_json }} # Whether to dynamically generate thumbnails if needed. - dynamic_thumbnails: {{ matrix_dendrite_mediaapi_dynamic_thumbnails|to_json }} + dynamic_thumbnails: {{ matrix_dendrite_media_api_dynamic_thumbnails|to_json }} # The maximum number of simultaneous thumbnail generators to run. - max_thumbnail_generators: {{ matrix_dendrite_mediaapi_max_thumbnails_generators|to_json }} + max_thumbnail_generators: {{ matrix_dendrite_media_api_max_thumbnails_generators|to_json }} # A list of thumbnail sizes to be generated for media content. thumbnail_sizes: @@ -355,7 +355,7 @@ sync_api: # Configuration for the full-text search engine. search: # Whether or not search is enabled. - enabled: {{ matrix_dendrite_syncapi_search_enabled|to_json }} + enabled: {{ matrix_dendrite_sync_api_search_enabled|to_json }} # The path where the search index will be created in. index_path: "/matrix-media-store-parent/searchindex" # The language most likely to be used on the server - used when indexing, to From 4c1db32ef920f8fa154868cd424782c234622cf4 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Mar 2023 08:52:15 +0200 Subject: [PATCH 329/382] Rename some Dendrite variables to improve consistency --- group_vars/matrix_servers | 12 ++++++------ roles/custom/matrix-dendrite/defaults/main.yml | 14 +++++++------- .../matrix-dendrite/tasks/validate_config.yml | 7 +++++++ .../templates/dendrite/dendrite.yaml.j2 | 14 +++++++------- 4 files changed, 27 insertions(+), 20 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 60c8349e..18ad756c 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2718,15 +2718,15 @@ devture_postgres_managed_databases_auto: | }] if (matrix_synapse_enabled and matrix_synapse_database_host == devture_postgres_connection_hostname) else []) + ([{ - 'name': matrix_dendrite_federationapi_database, + 'name': matrix_dendrite_federation_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_keyserver_database, + 'name': matrix_dendrite_key_server_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_mediaapi_database, + 'name': matrix_dendrite_media_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ @@ -2734,15 +2734,15 @@ devture_postgres_managed_databases_auto: | 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_syncapi_database, + 'name': matrix_dendrite_sync_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_userapi_database, + 'name': matrix_dendrite_user_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_pushserver_database, + 'name': matrix_dendrite_push_server_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 758a56e8..35793f49 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -150,13 +150,13 @@ matrix_dendrite_database_str: "postgresql://{{ matrix_dendrite_database_user }}: matrix_dendrite_database_hostname: '' matrix_dendrite_database_user: "dendrite" matrix_dendrite_database_password: "itsasecret" -matrix_dendrite_federationapi_database: "dendrite_federationapi" -matrix_dendrite_keyserver_database: "dendrite_keyserver" -matrix_dendrite_mediaapi_database: "dendrite_mediaapi" +matrix_dendrite_federation_api_database: "dendrite_federationapi" +matrix_dendrite_key_server_database: "dendrite_keyserver" +matrix_dendrite_media_api_database: "dendrite_mediaapi" matrix_dendrite_room_database: "dendrite_room" -matrix_dendrite_syncapi_database: "dendrite_syncapi" -matrix_dendrite_userapi_database: "dendrite_userapi" -matrix_dendrite_pushserver_database: "dendrite_pushserver" +matrix_dendrite_sync_api_database: "dendrite_syncapi" +matrix_dendrite_user_api_database: "dendrite_userapi" +matrix_dendrite_push_server_database: "dendrite_pushserver" matrix_dendrite_mscs_database: "dendrite_mscs" matrix_dendrite_client_api_turn_uris: [] @@ -206,7 +206,7 @@ matrix_dendrite_configuration_extension: "{{ matrix_dendrite_configuration_exten # You most likely don't need to touch this variable. Instead, see `matrix_dendrite_configuration_yaml`. matrix_dendrite_configuration: "{{ matrix_dendrite_configuration_yaml | from_yaml | combine(matrix_dendrite_configuration_extension, recursive=True) }}" -matrix_dendrite_userapi_auto_join_rooms: [] +matrix_dendrite_user_api_auto_join_rooms: [] # statistics reporting configuration. These statistics contain the server # name, number of active users and some information on your deployment config. diff --git a/roles/custom/matrix-dendrite/tasks/validate_config.yml b/roles/custom/matrix-dendrite/tasks/validate_config.yml index 53c17498..1f1b3e41 100644 --- a/roles/custom/matrix-dendrite/tasks/validate_config.yml +++ b/roles/custom/matrix-dendrite/tasks/validate_config.yml @@ -27,3 +27,10 @@ - {'old': 'matrix_dendrite_rate_limiting_enabled', 'new': 'matrix_dendrite_client_api_rate_limiting_enabled'} - {'old': 'matrix_dendrite_rate_limiting_threshold', 'new': 'matrix_dendrite_client_api_rate_limiting_threshold'} - {'old': 'matrix_dendrite_rate_limiting_cooloff_ms', 'new': 'matrix_dendrite_client_api_rate_limiting_cooloff_ms'} + - {'old': 'matrix_dendrite_mediaapi_database', 'new': 'matrix_dendrite_media_api_database'} + - {'old': 'matrix_dendrite_syncapi_database', 'new': 'matrix_dendrite_sync_api_database'} + - {'old': 'matrix_dendrite_userapi_database', 'new': 'matrix_dendrite_user_api_database'} + - {'old': 'matrix_dendrite_userapi_auto_join_rooms', 'new': 'matrix_dendrite_user_api_auto_join_rooms'} + - {'old': 'matrix_dendrite_federationapi_database', 'new': 'matrix_dendrite_federation_api_database'} + - {'old': 'matrix_dendrite_pushserver_database', 'new': 'matrix_dendrite_push_server_database'} + - {'old': 'matrix_dendrite_keyserver_database', 'new': 'matrix_dendrite_key_server_database'} diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index 1b31584d..37d0064b 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -223,7 +223,7 @@ federation_api: external_api: listen: http://0.0.0.0:8072 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_federationapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_federation_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -266,7 +266,7 @@ key_server: listen: http://0.0.0.0:7779 connect: http://key_server:7779 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_keyserver_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_key_server_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -279,7 +279,7 @@ media_api: external_api: listen: http://0.0.0.0:8074 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_mediaapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_media_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -342,7 +342,7 @@ sync_api: external_api: listen: http://0.0.0.0:8073 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_syncapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_sync_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -376,7 +376,7 @@ user_api: listen: http://0.0.0.0:7781 connect: http://user_api:7781 account_database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_userapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_user_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -385,7 +385,7 @@ user_api: # is considered to be valid in milliseconds. # The default lifetime is 3600000ms (60 minutes). # openid_token_lifetime_ms: 3600000 - auto_join_rooms: {{ matrix_dendrite_userapi_auto_join_rooms | to_json }} + auto_join_rooms: {{ matrix_dendrite_user_api_auto_join_rooms | to_json }} # Not in dendrite-config.yaml, but is in build/docker/config/dendrite.yaml # Configuration for the Push Server API. @@ -394,7 +394,7 @@ push_server: listen: http://localhost:7782 connect: http://localhost:7782 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_pushserver_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_push_server_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 From 5cd3a6188e7c01630c17f121a611a992a9c3bb74 Mon Sep 17 00:00:00 2001 From: Catalan Lover <48515417+FSG-Cat@users.noreply.github.com> Date: Tue, 14 Mar 2023 17:39:41 +0100 Subject: [PATCH 330/382] Upgrade Synapse from 1.78.0 to 1.79.0 --- roles/custom/matrix-synapse/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 41ad0bc4..a5c7106f 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -4,7 +4,7 @@ matrix_synapse_enabled: true -matrix_synapse_version: v1.78.0 +matrix_synapse_version: v1.79.0 matrix_synapse_username: '' matrix_synapse_uid: '' From fab6c03609751919af9467b935c115a7cabd7ebc Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 14 Mar 2023 19:20:46 +0200 Subject: [PATCH 331/382] Allow some new endpoints to work on workers Related to: - https://github.com/matrix-org/synapse/commit/682d31c7023b6b7299e74bc631e4d2acc60f91ac - https://github.com/matrix-org/synapse/commit/ecbe0ddbe7c47e05bc27b39dc10a9c30eafd2960 - https://github.com/matrix-org/synapse/commit/f4fc83ac755b8b06ecab1a31592308b03f8d2a5e .. and https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2569 --- roles/custom/matrix-synapse/vars/main.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/roles/custom/matrix-synapse/vars/main.yml b/roles/custom/matrix-synapse/vars/main.yml index 46277c87..faaf6ed6 100644 --- a/roles/custom/matrix-synapse/vars/main.yml +++ b/roles/custom/matrix-synapse/vars/main.yml @@ -163,7 +163,9 @@ matrix_synapse_workers_generic_worker_endpoints: - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$ - ^/_matrix/client/v1/rooms/.*/timestamp_to_event$ + - ^/_matrix/client/(api/v1|r0|v3|unstable/.*)/rooms/.*/aliases - ^/_matrix/client/(api/v1|r0|v3|unstable)/search$ + - ^/_matrix/client/(r0|v3|unstable)/user/.*/filter(/|$) # Encryption requests # Note that ^/_matrix/client/(r0|v3|unstable)/keys/upload/ requires `worker_main_http_uri` @@ -184,6 +186,7 @@ matrix_synapse_workers_generic_worker_endpoints: - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/join/ + - ^/_matrix/client/(api/v1|r0|v3|unstable)/knock/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/ # Start of intentionally-ignored-endpoints From a63860ac750e8aa87e12b06527dabdaa9bb436b7 Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Tue, 14 Mar 2023 14:10:16 -0400 Subject: [PATCH 332/382] fix typo --- .../custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index 37d0064b..89821de1 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -296,7 +296,7 @@ media_api: dynamic_thumbnails: {{ matrix_dendrite_media_api_dynamic_thumbnails|to_json }} # The maximum number of simultaneous thumbnail generators to run. - max_thumbnail_generators: {{ matrix_dendrite_media_api_max_thumbnails_generators|to_json }} + max_thumbnail_generators: {{ matrix_dendrite_media_api_max_thumbnail_generators|to_json }} # A list of thumbnail sizes to be generated for media content. thumbnail_sizes: From 99cc5e4b2f7ba38c22e50a3e8c8be0b1f3164b44 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 15 Mar 2023 14:14:02 +0200 Subject: [PATCH 333/382] Fix file name typo --- docs/installing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installing.md b/docs/installing.md index fccc1caf..c85ce1e6 100644 --- a/docs/installing.md +++ b/docs/installing.md @@ -17,7 +17,7 @@ Here are some playbook tags that you should be familiar with: - `install-all` - like `setup-all`, but skips uninstallation tasks. Useful for maintaining your setup quickly when its components remain unchanged. If you adjust your `vars.yml` to remove components, you'd need to run `setup-all` though, or these components will still remain installed -- `setup-SERVICE` (e.g. `setup-bot-postmoogle`) - runs the setup tasks only for a given role, but does not start/restart services. You can discover these additional tags in each role (`roles/*/main.yml`). Running per-component setup tasks is **not recommended**, as components sometimes depend on each other and running just the setup tasks for a given component may not be enough. For example, setting up the [mautrix-telegram bridge](configuring-playbook-bridge-mautrix-telegram.md), in addition to the `setup-mautrix-telegram` tag, requires database changes (the `setup-postgres` tag) as well as reverse-proxy changes (the `setup-nginx-proxy` tag). +- `setup-SERVICE` (e.g. `setup-bot-postmoogle`) - runs the setup tasks only for a given role, but does not start/restart services. You can discover these additional tags in each role (`roles/**/tasks/main.yml`). Running per-component setup tasks is **not recommended**, as components sometimes depend on each other and running just the setup tasks for a given component may not be enough. For example, setting up the [mautrix-telegram bridge](configuring-playbook-bridge-mautrix-telegram.md), in addition to the `setup-mautrix-telegram` tag, requires database changes (the `setup-postgres` tag) as well as reverse-proxy changes (the `setup-nginx-proxy` tag). - `install-SERVICE` (e.g. `install-bot-postmoogle`) - like `setup-SERVICE`, but skips uninstallation tasks. See `install-all` above for additional information. From eaa3be282558e958e01a67bba7b85b522d6263f6 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Wed, 15 Mar 2023 20:07:51 +0000 Subject: [PATCH 334/382] Update element 1.11.24 -> 1.11.25 --- roles/custom/matrix-client-element/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index 8e24ca66..97207285 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -10,7 +10,7 @@ matrix_client_element_container_image_self_build_repo: "https://github.com/vecto # - https://github.com/vector-im/element-web/issues/19544 matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}" -matrix_client_element_version: v1.11.24 +matrix_client_element_version: v1.11.25 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" From 9afc46de72113d5d44b6057b502978b60bd15d72 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Mar 2023 10:06:52 +0200 Subject: [PATCH 335/382] Update matrix_bot_chatgpt_matrix_blacklist/matrix_bot_chatgpt_matrix_whitelist comment --- roles/custom/matrix-bot-chatgpt/defaults/main.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index 13b25a00..49947e17 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -54,6 +54,9 @@ matrix_bot_chatgpt_matrix_default_prefix: "!chatgpt " # MATRIX_DEFAULT_PREFIX= matrix_bot_chatgpt_matrix_default_prefix_reply: false # MATRIX_DEFAULT_PREFIX_REPLY= # Matrix Access Control (optional) +# +# Lists of space-separated entries. Example: +# matrix_bot_chatgpt_matrix_whitelist: ":{{ matrix_domain }} :second-domain.com :third-domain.com" matrix_bot_chatgpt_matrix_blacklist: '' matrix_bot_chatgpt_matrix_whitelist: ':{{ matrix_domain }}' From a55c6b1b5a573fd67ad5ad77a5458c12c3739968 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Mar 2023 10:07:26 +0200 Subject: [PATCH 336/382] Remove non-functional variable matrix_bot_chatgpt_loglevel It wasn't used at all. It doesn't seem like there's a way to control the log level, judging by: - https://github.com/matrixgpt/matrix-chatgpt-bot/blob/8f9eea4114e74597dd7cfcc9454bf327ec32c2a3/.env.example - https://github.com/matrixgpt/matrix-chatgpt-bot/blob/8f9eea4114e74597dd7cfcc9454bf327ec32c2a3/src/env.ts --- roles/custom/matrix-bot-chatgpt/defaults/main.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index 49947e17..f34eb1e1 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -76,9 +76,6 @@ matrix_bot_chatgpt_matrix_rich_text: true # MATRIX_RICH_TEXT=true # - @*:another.com # matrix_bot_chatgpt_admins: "{{ [matrix_admin] if matrix_admin else [] }}" -# Log level -# matrix_bot_chatgpt_loglevel: 'INFO' - # Additional environment variables to pass to the chatgpt container # # Example: From 636142c4e0aae733373170087e6bf4681b03c09d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Mar 2023 10:19:23 +0200 Subject: [PATCH 337/382] matrix-bot-chatgpt role refactoring --- .../matrix-bot-chatgpt/defaults/main.yml | 20 +++--- .../matrix-bot-chatgpt/tasks/install.yml | 63 +++++++++++++++++++ .../custom/matrix-bot-chatgpt/tasks/main.yml | 4 +- .../tasks/setup_install.yml | 63 ------------------- .../{setup_uninstall.yml => uninstall.yml} | 0 .../tasks/validate_config.yml | 15 ++++- .../systemd/matrix-bot-chatgpt.service.j2 | 6 +- 7 files changed, 96 insertions(+), 75 deletions(-) create mode 100644 roles/custom/matrix-bot-chatgpt/tasks/install.yml delete mode 100644 roles/custom/matrix-bot-chatgpt/tasks/setup_install.yml rename roles/custom/matrix-bot-chatgpt/tasks/{setup_uninstall.yml => uninstall.yml} (100%) diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml index f34eb1e1..16ad8588 100644 --- a/roles/custom/matrix-bot-chatgpt/defaults/main.yml +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -4,19 +4,22 @@ matrix_bot_chatgpt_enabled: true +matrix_bot_chatgpt_version: 3.0.0 + matrix_bot_chatgpt_container_image_self_build: false -matrix_bot_chatgpt_docker_repo: "https://github.com/matrixgpt/matrix-chatgpt-bot" -matrix_bot_chatgpt_docker_repo_version: "{{ 'latest' if matrix_bot_chatgpt_version == 'latest' else matrix_bot_chatgpt_version }}" -matrix_bot_chatgpt_docker_src_files_path: "{{ matrix_base_data_path }}/chatgpt/docker-src" +matrix_bot_chatgpt_container_image_self_build_repo: "https://github.com/matrixgpt/matrix-chatgpt-bot" +matrix_bot_chatgpt_container_image_self_build_repo_version: "{{ 'main' if matrix_bot_chatgpt_version == 'latest' else matrix_bot_chatgpt_version }}" -matrix_bot_chatgpt_version: 3.0.0 -matrix_bot_chatgpt_docker_image: "{{ matrix_bot_chatgpt_docker_image_name_prefix }}matrixgpt/matrix-chatgpt-bot:{{ matrix_bot_chatgpt_version }}" -matrix_bot_chatgpt_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_chatgpt_container_image_self_build else 'ghcr.io/' }}" -matrix_bot_chatgpt_docker_image_force_pull: "{{ matrix_bot_chatgpt_docker_image.endswith(':latest') }}" +matrix_bot_chatgpt_container_image: "{{ matrix_bot_chatgpt_container_image_name_prefix }}matrixgpt/matrix-chatgpt-bot:{{ matrix_bot_chatgpt_container_image_tag }}" +matrix_bot_chatgpt_container_image_name_prefix: "{{ 'localhost/' if matrix_bot_chatgpt_container_image_self_build else matrix_bot_chatgpt_container_image_registry_prefix }}" +matrix_bot_chatgpt_container_image_tag: "{{ matrix_bot_chatgpt_version }}" +matrix_bot_chatgpt_container_image_force_pull: "{{ matrix_bot_chatgpt_container_image.endswith(':latest') }}" +matrix_bot_chatgpt_container_image_registry_prefix: ghcr.io/ matrix_bot_chatgpt_base_path: "{{ matrix_base_data_path }}/chatgpt" matrix_bot_chatgpt_config_path: "{{ matrix_bot_chatgpt_base_path }}/config" matrix_bot_chatgpt_data_path: "{{ matrix_bot_chatgpt_base_path }}/data" +matrix_bot_chatgpt_container_src_path: "{{ matrix_bot_chatgpt_base_path }}/container-src" # A list of extra arguments to pass to the container matrix_bot_chatgpt_container_extra_arguments: [] @@ -78,6 +81,9 @@ matrix_bot_chatgpt_matrix_rich_text: true # MATRIX_RICH_TEXT=true # Additional environment variables to pass to the chatgpt container # +# You can discover additional environment variables from: +# https://github.com/matrixgpt/matrix-chatgpt-bot/blob/main/src/env.ts +# # Example: # matrix_bot_chatgpt_environment_variables_extension: | # chatgpt_TEXT_DONE=Done diff --git a/roles/custom/matrix-bot-chatgpt/tasks/install.yml b/roles/custom/matrix-bot-chatgpt/tasks/install.yml new file mode 100644 index 00000000..ef31df06 --- /dev/null +++ b/roles/custom/matrix-bot-chatgpt/tasks/install.yml @@ -0,0 +1,63 @@ +--- + +- name: Ensure chatgpt paths exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - {path: "{{ matrix_bot_chatgpt_config_path }}", when: true} + - {path: "{{ matrix_bot_chatgpt_data_path }}", when: true} + - {path: "{{ matrix_bot_chatgpt_container_src_path }}", when: matrix_bot_chatgpt_container_image_self_build} + when: "item.when | bool" + +- name: Ensure chatgpt environment variables file created + ansible.builtin.template: + src: "{{ role_path }}/templates/env.j2" + dest: "{{ matrix_bot_chatgpt_config_path }}/env" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + mode: 0640 + +- name: Ensure chatgpt container image is pulled + community.docker.docker_image: + name: "{{ matrix_bot_chatgpt_container_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_bot_chatgpt_container_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_chatgpt_container_image_force_pull }}" + when: "not matrix_bot_chatgpt_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- when: matrix_bot_chatgpt_container_image_self_build | bool + block: + - name: Ensure chatgpt repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_bot_chatgpt_container_image_self_build_repo }}" + version: "{{ matrix_bot_chatgpt_container_image_self_build_repo_version }}" + dest: "{{ matrix_bot_chatgpt_container_src_path }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_bot_chatgpt_git_pull_results + + - name: Ensure chatgpt container image is built + community.docker.docker_image: + name: "{{ matrix_bot_chatgpt_container_image }}" + source: build + force_source: "{{ matrix_bot_chatgpt_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_bot_chatgpt_container_src_path }}" + pull: true + +- name: Ensure matrix-bot-chatgpt.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-bot-chatgpt.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-chatgpt.service" + mode: 0644 diff --git a/roles/custom/matrix-bot-chatgpt/tasks/main.yml b/roles/custom/matrix-bot-chatgpt/tasks/main.yml index 1086a62d..3f5a7b7e 100644 --- a/roles/custom/matrix-bot-chatgpt/tasks/main.yml +++ b/roles/custom/matrix-bot-chatgpt/tasks/main.yml @@ -11,11 +11,11 @@ ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - when: matrix_bot_chatgpt_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" - tags: - setup-all - setup-bot-chatgpt block: - when: not matrix_bot_chatgpt_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" diff --git a/roles/custom/matrix-bot-chatgpt/tasks/setup_install.yml b/roles/custom/matrix-bot-chatgpt/tasks/setup_install.yml deleted file mode 100644 index 5ae5ea95..00000000 --- a/roles/custom/matrix-bot-chatgpt/tasks/setup_install.yml +++ /dev/null @@ -1,63 +0,0 @@ ---- -- name: Ensure chatgpt paths exist - ansible.builtin.file: - path: "{{ item.path }}" - state: directory - mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - with_items: - - {path: "{{ matrix_bot_chatgpt_config_path }}", when: true} - - {path: "{{ matrix_bot_chatgpt_data_path }}", when: true} - - {path: "{{ matrix_bot_chatgpt_docker_src_files_path }}", when: matrix_bot_chatgpt_container_image_self_build} - when: "item.when | bool" - -- name: Ensure chatgpt environment variables file created - ansible.builtin.template: - src: "{{ role_path }}/templates/env.j2" - dest: "{{ matrix_bot_chatgpt_config_path }}/env" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - mode: 0640 - -- name: Ensure chatgpt image is pulled - community.docker.docker_image: - name: "{{ matrix_bot_chatgpt_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_bot_chatgpt_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_chatgpt_docker_image_force_pull }}" - when: "not matrix_bot_chatgpt_container_image_self_build | bool" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure chatgpt repository is present on self-build - ansible.builtin.git: - repo: "{{ matrix_bot_chatgpt_docker_repo }}" - version: "{{ matrix_bot_chatgpt_docker_repo_version }}" - dest: "{{ matrix_bot_chatgpt_docker_src_files_path }}" - force: "yes" - become: true - become_user: "{{ matrix_user_username }}" - register: matrix_bot_chatgpt_git_pull_results - when: "matrix_bot_chatgpt_container_image_self_build | bool" - -- name: Ensure chatgpt image is built - community.docker.docker_image: - name: "{{ matrix_bot_chatgpt_docker_image }}" - source: build - force_source: "{{ matrix_bot_chatgpt_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" - build: - dockerfile: Dockerfile - path: "{{ matrix_bot_chatgpt_docker_src_files_path }}" - pull: true - when: "matrix_bot_chatgpt_container_image_self_build | bool" - -- name: Ensure matrix-bot-chatgpt.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-bot-chatgpt.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-chatgpt.service" - mode: 0644 - register: matrix_bot_chatgpt_systemd_service_result diff --git a/roles/custom/matrix-bot-chatgpt/tasks/setup_uninstall.yml b/roles/custom/matrix-bot-chatgpt/tasks/uninstall.yml similarity index 100% rename from roles/custom/matrix-bot-chatgpt/tasks/setup_uninstall.yml rename to roles/custom/matrix-bot-chatgpt/tasks/uninstall.yml diff --git a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml index ecc2b8d2..c6771823 100644 --- a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml +++ b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml @@ -9,7 +9,6 @@ - {'name': 'matrix_bot_chatgpt_openai_api_key', when: true} - {'name': 'matrix_bot_chatgpt_matrix_bot_username', when: true} - - name: Fail if OpenAI configuration not up-to-date. ansible.builtin.fail: msg: >- @@ -20,3 +19,17 @@ - {'name': 'matrix_bot_chatgpt_openai_email'} - {'name': 'matrix_bot_chatgpt_openai_password'} - {'name': 'matrix_bot_chatgpt_openai_login_type'} + +- name: (Deprecation) Catch and report renamed ChatGPT settings + ansible.builtin.fail: + msg: >- + Your configuration contains a variable, which now has a different name. + Please change your configuration to rename the variable (`{{ item.old }}` -> `{{ item.new }}`). + when: "item.old in vars" + with_items: + - {'old': 'matrix_bot_chatgpt_docker_image', 'new': 'matrix_bot_chatgpt_container_image'} + - {'old': 'matrix_bot_chatgpt_docker_image_name_prefix', 'new': 'matrix_bot_chatgpt_container_image_name_prefix'} + - {'old': 'matrix_bot_chatgpt_docker_image_force_pull', 'new': 'matrix_bot_chatgpt_container_image_force_pull'} + - {'old': 'matrix_bot_chatgpt_docker_repo', 'new': 'matrix_bot_chatgpt_container_image_self_build_repo'} + - {'old': 'matrix_bot_chatgpt_docker_repo_version', 'new': 'matrix_bot_chatgpt_container_image_self_build_repo_version'} + - {'old': 'matrix_bot_chatgpt_docker_src_files_path', 'new': 'matrix_bot_chatgpt_container_src_path'} diff --git a/roles/custom/matrix-bot-chatgpt/templates/systemd/matrix-bot-chatgpt.service.j2 b/roles/custom/matrix-bot-chatgpt/templates/systemd/matrix-bot-chatgpt.service.j2 index 8e4a5020..c8fa06c1 100644 --- a/roles/custom/matrix-bot-chatgpt/templates/systemd/matrix-bot-chatgpt.service.j2 +++ b/roles/custom/matrix-bot-chatgpt/templates/systemd/matrix-bot-chatgpt.service.j2 @@ -16,7 +16,9 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-chatgpt 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-chatgpt 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-bot-chatgpt \ +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run \ + --rm \ + --name=matrix-bot-chatgpt \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ @@ -29,7 +31,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {% for arg in matrix_bot_chatgpt_container_extra_arguments %} {{ arg }} \ {% endfor %} - {{ matrix_bot_chatgpt_docker_image }} + {{ matrix_bot_chatgpt_container_image }} ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-chatgpt 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-chatgpt 2>/dev/null || true' From 2b49ae79ca22f9ef434eb8ca4b7d229ba72a5958 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Mar 2023 11:23:22 +0200 Subject: [PATCH 338/382] Use just for Postgres maintenance command examples --- docs/importing-postgres.md | 14 +++++++------- docs/maintenance-postgres.md | 6 ++++-- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/importing-postgres.md b/docs/importing-postgres.md index a44afdcf..150164e7 100644 --- a/docs/importing-postgres.md +++ b/docs/importing-postgres.md @@ -20,18 +20,18 @@ Before doing the actual import, **you need to upload your Postgres dump file to ## Importing -To import, run this command (make sure to replace `` with a file path on your server): +To import, run this command (make sure to replace `SERVER_PATH_TO_POSTGRES_DUMP_FILE` with a file path on your server): ```sh -ansible-playbook -i inventory/hosts setup.yml \ ---extra-vars='server_path_postgres_dump= postgres_default_import_database=matrix' \ ---tags=import-postgres +just run-tags import-postgres \ +--extra-vars=server_path_postgres_dump=SERVER_PATH_TO_POSTGRES_DUMP_FILE \ +--extra-vars=postgres_default_import_database=main ``` **Notes**: -- `` must be a file path to a Postgres dump file on the server (not on your local machine!) -- `postgres_default_import_database` defaults to `matrix`, which is useful for importing multiple databases (for dumps made with `pg_dumpall`). If you're importing a single database (e.g. `synapse`), consider changing `postgres_default_import_database` accordingly +- `SERVER_PATH_TO_POSTGRES_DUMP_FILE` must be a file path to a Postgres dump file on the server (not on your local machine!) +- `postgres_default_import_database` defaults to `main`, which is useful for importing multiple databases (for dumps made with `pg_dumpall`). If you're importing a single database (e.g. `synapse`), consider changing `postgres_default_import_database` accordingly ## Troubleshooting @@ -86,7 +86,7 @@ In this case you can use the command suggested in the import task to clear the d # systemctl start matrix-postgres ``` -Now on your local machine run `ansible-playbook -i inventory/hosts setup.yml --tags=setup-postgres` to prepare the database roles etc. +Now on your local machine run `just run-tags setup-postgres` to prepare the database roles etc. If not, you probably get this error. `synapse` is the correct table owner, but the role is missing in database. ``` diff --git a/docs/maintenance-postgres.md b/docs/maintenance-postgres.md index f2ca907d..cc8898a2 100644 --- a/docs/maintenance-postgres.md +++ b/docs/maintenance-postgres.md @@ -41,7 +41,7 @@ To perform a `FULL` Postgres [VACUUM](https://www.postgresql.org/docs/current/sq Example: ```bash -ansible-playbook -i inventory/hosts setup.yml --tags=run-postgres-vacuum,start +just run-tags run-postgres-vacuum,start ``` **Note**: this will automatically stop Synapse temporarily and restart it later. You'll also need plenty of available disk space in your Postgres data directory (usually `/matrix/postgres/data`). @@ -78,7 +78,9 @@ Upgrades must be performed manually. This playbook can upgrade your existing Postgres setup with the following command: - ansible-playbook -i inventory/hosts setup.yml --tags=upgrade-postgres +```sh +just run-tags upgrade-postgres +``` **Warning: If you're using Borg Backup keep in mind that there is no official Postgres 15 support yet.** From d6975e4ab8e3fd63cf325caae4b7fcd8f7e307f8 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Mar 2023 14:57:40 +0200 Subject: [PATCH 339/382] Upgrade com.devture.ansible.role.postgres --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index e841cb4e..6690f7c0 100644 --- a/requirements.yml +++ b/requirements.yml @@ -19,7 +19,7 @@ version: ff2fd42e1c1a9e28e3312bbd725395f9c2fc7f16 - src: git+https://github.com/devture/com.devture.ansible.role.postgres.git - version: a1bb78d194434b38005f3a9e623bfa4b2c06c7bc + version: 38764398bf82b06a1736c3bfedc71dfd229e4b52 - src: git+https://github.com/devture/com.devture.ansible.role.postgres_backup.git version: 8e9ec48a09284c84704d7a2dce17da35f181574d From ee4bac1988e2ae29e9e89fbc907bfaaca7062df7 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 16 Mar 2023 18:29:11 +0200 Subject: [PATCH 340/382] Upgrade matrix-corporal (2.5.1 -> 2.5.2) --- roles/custom/matrix-corporal/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-corporal/defaults/main.yml b/roles/custom/matrix-corporal/defaults/main.yml index 9f6da78a..2b703bdd 100644 --- a/roles/custom/matrix-corporal/defaults/main.yml +++ b/roles/custom/matrix-corporal/defaults/main.yml @@ -23,7 +23,7 @@ matrix_corporal_container_extra_arguments: [] # List of systemd services that matrix-corporal.service depends on matrix_corporal_systemd_required_services_list: ['docker.service'] -matrix_corporal_version: 2.5.1 +matrix_corporal_version: 2.5.2 matrix_corporal_docker_image: "{{ matrix_corporal_docker_image_name_prefix }}devture/matrix-corporal:{{ matrix_corporal_docker_image_tag }}" matrix_corporal_docker_image_name_prefix: "{{ 'localhost/' if matrix_corporal_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_corporal_docker_image_tag: "{{ matrix_corporal_version }}" # for backward-compatibility From ff6e7f0ac5063aa9e67ccab53ef79a9b66e5702e Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Thu, 16 Mar 2023 21:03:55 +0000 Subject: [PATCH 341/382] Update borgmatic 1.7.8 -> 1.7.9 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 6690f7c0..77f8186b 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.11.1-2 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git - version: v1.2.3-1.7.8-0 + version: v1.2.3-1.7.9-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git version: v9.4.3-0 From 56b0706863f7c1f2851c1dd08faf3b4fa3128eda Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Fri, 17 Mar 2023 08:27:25 +0200 Subject: [PATCH 342/382] Fix default database name For backward-compatibility reasons, this playbook uses `matrix` as its main and default database. --- docs/importing-postgres.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/importing-postgres.md b/docs/importing-postgres.md index 150164e7..3c693578 100644 --- a/docs/importing-postgres.md +++ b/docs/importing-postgres.md @@ -25,13 +25,13 @@ To import, run this command (make sure to replace `SERVER_PATH_TO_POSTGRES_DUMP_ ```sh just run-tags import-postgres \ --extra-vars=server_path_postgres_dump=SERVER_PATH_TO_POSTGRES_DUMP_FILE \ ---extra-vars=postgres_default_import_database=main +--extra-vars=postgres_default_import_database=matrix ``` **Notes**: - `SERVER_PATH_TO_POSTGRES_DUMP_FILE` must be a file path to a Postgres dump file on the server (not on your local machine!) -- `postgres_default_import_database` defaults to `main`, which is useful for importing multiple databases (for dumps made with `pg_dumpall`). If you're importing a single database (e.g. `synapse`), consider changing `postgres_default_import_database` accordingly +- `postgres_default_import_database` defaults to `matrix`, which is useful for importing multiple databases (for dumps made with `pg_dumpall`). If you're importing a single database (e.g. `synapse`), consider changing `postgres_default_import_database` accordingly ## Troubleshooting From 281bad125d7705bda60e6e98de5520daf53ee9cd Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 17 Mar 2023 20:01:57 +0000 Subject: [PATCH 343/382] Update mautrix-discord 0.1.1 -> 0.2.0 --- roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml index 036b4060..1bbcb02b 100644 --- a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_discord_container_image_self_build: false matrix_mautrix_discord_container_image_self_build_repo: "https://mau.dev/mautrix/discord.git" matrix_mautrix_discord_container_image_self_build_branch: "{{ 'main' if matrix_mautrix_discord_version == 'latest' else matrix_mautrix_discord_version }}" -matrix_mautrix_discord_version: v0.1.1 +matrix_mautrix_discord_version: v0.2.0 # See: https://mau.dev/mautrix/discord/container_registry matrix_mautrix_discord_docker_image: "{{ matrix_mautrix_discord_docker_image_name_prefix }}mautrix/discord:{{ matrix_mautrix_discord_version }}" matrix_mautrix_discord_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_discord_container_image_self_build else 'dock.mau.dev/' }}" From e6552a9c3b537f1ca3aad03e8d02e0a447208c28 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 17 Mar 2023 20:03:17 +0000 Subject: [PATCH 344/382] Update mautrix-whatsapp 0.8.2 -> 0.8.3 --- roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml index 04c24295..ac5796ba 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_whatsapp_container_image_self_build: false matrix_mautrix_whatsapp_container_image_self_build_repo: "https://mau.dev/mautrix/whatsapp.git" matrix_mautrix_whatsapp_container_image_self_build_branch: "{{ 'master' if matrix_mautrix_whatsapp_version == 'latest' else matrix_mautrix_whatsapp_version }}" -matrix_mautrix_whatsapp_version: v0.8.2 +matrix_mautrix_whatsapp_version: v0.8.3 # See: https://mau.dev/mautrix/whatsapp/container_registry matrix_mautrix_whatsapp_docker_image: "{{ matrix_mautrix_whatsapp_docker_image_name_prefix }}mautrix/whatsapp:{{ matrix_mautrix_whatsapp_version }}" matrix_mautrix_whatsapp_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_whatsapp_container_image_self_build else 'dock.mau.dev/' }}" From 7ef804f62f107f420e853ecc9ebe416264521d17 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 17 Mar 2023 20:05:38 +0000 Subject: [PATCH 345/382] Update hookshot 2.7.0 -> 3.0.0 Warning! Potentially breaking change: https://github.com/matrix-org/matrix-hookshot/releases/tag/3.0.0 --- roles/custom/matrix-bridge-hookshot/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-hookshot/defaults/main.yml b/roles/custom/matrix-bridge-hookshot/defaults/main.yml index 3b65b92a..db71da79 100644 --- a/roles/custom/matrix-bridge-hookshot/defaults/main.yml +++ b/roles/custom/matrix-bridge-hookshot/defaults/main.yml @@ -10,7 +10,7 @@ matrix_hookshot_container_image_self_build: false matrix_hookshot_container_image_self_build_repo: "https://github.com/matrix-org/matrix-hookshot.git" matrix_hookshot_container_image_self_build_branch: "{{ 'main' if matrix_hookshot_version == 'latest' else matrix_hookshot_version }}" -matrix_hookshot_version: 2.7.0 +matrix_hookshot_version: 3.0.0 matrix_hookshot_docker_image: "{{ matrix_hookshot_docker_image_name_prefix }}halfshot/matrix-hookshot:{{ matrix_hookshot_version }}" matrix_hookshot_docker_image_name_prefix: "{{ 'localhost/' if matrix_hookshot_container_image_self_build else matrix_container_global_registry_prefix }}" From f1c0321a8ce94f5e86c8b3fc0aecc9c4dac03f9a Mon Sep 17 00:00:00 2001 From: array-in-a-matrix Date: Sat, 18 Mar 2023 03:22:30 -0400 Subject: [PATCH 346/382] add relay api database for dendrite --- group_vars/matrix_servers | 4 ++++ roles/custom/matrix-dendrite/defaults/main.yml | 1 + roles/custom/matrix-dendrite/tasks/validate_config.yml | 1 + .../matrix-dendrite/templates/dendrite/dendrite.yaml.j2 | 6 ++++++ 4 files changed, 12 insertions(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 18ad756c..6be30d7e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2741,6 +2741,10 @@ devture_postgres_managed_databases_auto: | 'name': matrix_dendrite_user_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, + },{ + 'name': matrix_dendrite_relay_api_database, + 'username': matrix_dendrite_relay_api_user, + 'password': matrix_dendrite_relay_api_password, },{ 'name': matrix_dendrite_push_server_database, 'username': matrix_dendrite_database_user, diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 35793f49..41cd35ed 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -157,6 +157,7 @@ matrix_dendrite_room_database: "dendrite_room" matrix_dendrite_sync_api_database: "dendrite_syncapi" matrix_dendrite_user_api_database: "dendrite_userapi" matrix_dendrite_push_server_database: "dendrite_pushserver" +matrix_dendrite_relay_api_database: "dendrite_relayapi" matrix_dendrite_mscs_database: "dendrite_mscs" matrix_dendrite_client_api_turn_uris: [] diff --git a/roles/custom/matrix-dendrite/tasks/validate_config.yml b/roles/custom/matrix-dendrite/tasks/validate_config.yml index 1f1b3e41..0b1540bb 100644 --- a/roles/custom/matrix-dendrite/tasks/validate_config.yml +++ b/roles/custom/matrix-dendrite/tasks/validate_config.yml @@ -33,4 +33,5 @@ - {'old': 'matrix_dendrite_userapi_auto_join_rooms', 'new': 'matrix_dendrite_user_api_auto_join_rooms'} - {'old': 'matrix_dendrite_federationapi_database', 'new': 'matrix_dendrite_federation_api_database'} - {'old': 'matrix_dendrite_pushserver_database', 'new': 'matrix_dendrite_push_server_database'} + - {'old': 'matrix_dendrite_relayapi_database', 'new': 'matrix_dendrite_relay_api_database'} - {'old': 'matrix_dendrite_keyserver_database', 'new': 'matrix_dendrite_key_server_database'} diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index 89821de1..3c1e56e5 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -399,6 +399,12 @@ push_server: max_idle_conns: 2 conn_max_lifetime: -1 +# +# +relay_api: + database: + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_relay_api_database }}?sslmode=disable + # Configuration for Opentracing. # See https://github.com/matrix-org/dendrite/tree/master/docs/tracing for information on # how this works and how to set it up. From dd1712d45731675bb05bcf5e7c71922faeca62fd Mon Sep 17 00:00:00 2001 From: Array in a Matrix Date: Sat, 18 Mar 2023 03:43:12 -0400 Subject: [PATCH 347/382] fix typo i was sleepy lol --- group_vars/matrix_servers | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 6be30d7e..ef36a06c 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -2743,8 +2743,8 @@ devture_postgres_managed_databases_auto: | 'password': matrix_dendrite_database_password, },{ 'name': matrix_dendrite_relay_api_database, - 'username': matrix_dendrite_relay_api_user, - 'password': matrix_dendrite_relay_api_password, + 'username': matrix_dendrite_database_user, + 'password': matrix_dendrite_database_password, },{ 'name': matrix_dendrite_push_server_database, 'username': matrix_dendrite_database_user, From 88dc5e0de048bba8f4cbb0b5e1c3bc473b46c4dd Mon Sep 17 00:00:00 2001 From: Aine Date: Sat, 18 Mar 2023 10:26:29 +0200 Subject: [PATCH 348/382] migrate prometheus-node-exporter's var --- group_vars/matrix_servers | 2 ++ requirements.yml | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index ef36a06c..45a3990d 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -3447,7 +3447,9 @@ prometheus_node_exporter_base_path: "{{ matrix_base_data_path }}/prometheus-node prometheus_node_exporter_uid: "{{ matrix_user_uid }}" prometheus_node_exporter_gid: "{{ matrix_user_gid }}" +# _server_fqn is the old var, _hostname - the new one. Seamless migration prometheus_node_exporter_server_fqn: "{{ matrix_server_fqn_matrix }}" +prometheus_node_exporter_hostname: "{{ prometheus_node_exporter_server_fqn }}" prometheus_node_exporter_container_network: "{{ matrix_docker_network }}" diff --git a/requirements.yml b/requirements.yml index 77f8186b..9f5719c9 100644 --- a/requirements.yml +++ b/requirements.yml @@ -31,7 +31,7 @@ version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git - version: v1.5.0-3 + version: v1.5.0-4 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.11.1-2 From 8202acf74209111d859bb7db4d0b342a6642e12e Mon Sep 17 00:00:00 2001 From: Kabir Kwatra Date: Sat, 18 Mar 2023 10:17:29 +0000 Subject: [PATCH 349/382] fix(sliding-sync): use external syncv3 server url --- roles/custom/matrix-sliding-sync/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-sliding-sync/defaults/main.yml b/roles/custom/matrix-sliding-sync/defaults/main.yml index f66b3b61..fa6e65a4 100644 --- a/roles/custom/matrix-sliding-sync/defaults/main.yml +++ b/roles/custom/matrix-sliding-sync/defaults/main.yml @@ -76,7 +76,7 @@ matrix_sliding_sync_systemd_required_services_list: ["docker.service"] matrix_sliding_sync_systemd_wanted_services_list: [] # Controls the SYNCV3_SERVER environment variable -matrix_sliding_sync_environment_variable_syncv3_server: "{{ matrix_homeserver_container_url }}" +matrix_sliding_sync_environment_variable_syncv3_server: "{{ matrix_homeserver_url }}" # Controls the SYNCV3_SECRET environment variable matrix_sliding_sync_environment_variable_syncv3_secret: '' From b01e7b1ae59ea2602cd9f2da2c7142e97f41a695 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Sat, 18 Mar 2023 19:29:20 +0000 Subject: [PATCH 350/382] update ntfy 2.1.2 -> 2.2.0 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 9f5719c9..85b396b4 100644 --- a/requirements.yml +++ b/requirements.yml @@ -43,7 +43,7 @@ version: v9.4.3-0 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git - version: v2.1.2-0 + version: v2.2.0-0 - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.9-0 From 3e84e354a845df9863c67486ec7b2880cc435dd3 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 19 Mar 2023 08:12:37 +0200 Subject: [PATCH 351/382] Add "install-service" justfile command It should be noted that this cannot be used for the initial install of services which require a database or have other dependencies. Those would typically need to invoke the playbook with `--tags=install-postgres,install-SERVICE`, etc. The purpose of this shortcut is to easily rebuild and restart a single serice subsequently. For those cases, often times there's no need to reinitialize the database and other components and simply running a single component's tasks is enough. --- justfile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/justfile b/justfile index 4b6dfbfb..3b5c4de3 100644 --- a/justfile +++ b/justfile @@ -14,6 +14,10 @@ lint: # Runs the playbook with --tags=install-all,ensure-matrix-users-created,start and optional arguments install-all *extra_args: (run-tags "install-all,ensure-matrix-users-created,start" extra_args) +# Runs installation tasks for a single service +install-service service: + just --justfile {{ justfile() }} run --tags=install-{{ service }},start-group --extra-vars=group={{ service }} + # Runs the playbook with --tags=setup-all,ensure-matrix-users-created,start and optional arguments setup-all *extra_args: (run-tags "setup-all,ensure-matrix-users-created,start" extra_args) From 39f53e81d0d47d76a28787b9b05a7152cca90a16 Mon Sep 17 00:00:00 2001 From: Justin Croonenberghs Date: Sun, 19 Mar 2023 13:42:41 -0500 Subject: [PATCH 352/382] Update sliding sync doc for external DB users * fixes #2586 --- docs/configuring-playbook-sliding-sync-proxy.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/configuring-playbook-sliding-sync-proxy.md b/docs/configuring-playbook-sliding-sync-proxy.md index 49e0483a..a0eb36ba 100644 --- a/docs/configuring-playbook-sliding-sync-proxy.md +++ b/docs/configuring-playbook-sliding-sync-proxy.md @@ -40,6 +40,17 @@ matrix_sliding_sync_enabled: true After potentially [adjusting DNS records](#adjusting-dns-records) and configuring the playbook, run the [installation](installing.md) command again: `just install-all`. +### External databases + +Please note that, if your setup utilizes an external database, you must also establish configuration for the sliding sync proxy. Alter the defaults below to suit your configuration: + +```yaml +matrix_sliding_sync_database_username: 'matrix_sliding_sync' +matrix_sliding_sync_database_password: '' +matrix_sliding_sync_database_hostname: '' +matrix_sliding_sync_database_port: 5432 +matrix_sliding_sync_database_name: 'matrix_sliding_sync' +``` ## Usage From 220d80ac3a40b9ed5541299fbb283e4a1759254d Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 11:06:27 +0200 Subject: [PATCH 353/382] Move matrix-aux outside of this playbook --- group_vars/matrix_servers | 20 +++++ playbooks/matrix.yml | 3 +- requirements.yml | 4 + roles/custom/matrix-aux/defaults/main.yml | 81 ------------------- roles/custom/matrix-aux/tasks/main.yml | 9 --- roles/custom/matrix-aux/tasks/setup.yml | 20 ----- .../defaults/main.yml | 3 + .../tasks/validate_config.yml | 16 ++++ 8 files changed, 45 insertions(+), 111 deletions(-) delete mode 100644 roles/custom/matrix-aux/defaults/main.yml delete mode 100644 roles/custom/matrix-aux/tasks/main.yml delete mode 100644 roles/custom/matrix-aux/tasks/setup.yml diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 45a3990d..bdc2d123 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -37,6 +37,26 @@ matrix_playbook_ssl_enabled: "{{ matrix_playbook_ssl_retrieval_method in ['lets- # # ######################################################################## + +######################################################################## +# # +# aux # +# # +######################################################################## + +aux_directory_default_owner: "{{ matrix_user_username }}" +aux_directory_default_group: "{{ matrix_user_groupname }}" + +aux_file_default_owner: "{{ matrix_user_username }}" +aux_file_default_group: "{{ matrix_user_groupname }}" + +######################################################################## +# # +# /aux # +# # +######################################################################## + + ######################################################################## # # # base # diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index b8964aa5..93e9208f 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -112,7 +112,8 @@ - galaxy/ntfy - custom/matrix-nginx-proxy - custom/matrix-coturn - - custom/matrix-aux + + - role: galaxy/aux - role: galaxy/com.devture.ansible.role.postgres_backup diff --git a/requirements.yml b/requirements.yml index 85b396b4..ffcd86af 100644 --- a/requirements.yml +++ b/requirements.yml @@ -59,3 +59,7 @@ - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: v2.8.1-0 + +- src: git+https://github.com/mother-of-all-self-hosting/ansible-role-aux.git + name: aux + version: v1.0.0-0 diff --git a/roles/custom/matrix-aux/defaults/main.yml b/roles/custom/matrix-aux/defaults/main.yml deleted file mode 100644 index 4c1f8879..00000000 --- a/roles/custom/matrix-aux/defaults/main.yml +++ /dev/null @@ -1,81 +0,0 @@ ---- - -# matrix-aux is a role that manages auxiliary files and directories on your Matrix server. -# -# Certain components (like matrix-synapse, etc.) may sometimes require additional templates (email templates, privacy policies, etc.). -# This role allows such files to be managed by the playbook. -# -# Note that files and directories created via this role are not automatically made available for containers to use. -# If you use this role to put files in a directory that's already mounted into a container, -# you can access the files without additional work. -# Otherwise, you'd need to mount the file/directory to the container that needs it. -# Roles usually provide a `matrix_*_additional_volumes` or `matrix_*_container_extra_arguments` variable -# that you can use to mount an additional volume. - -# The default permission mode when creating directories using `matrix_aux_directory_definitions` -matrix_aux_directory_default_mode: '0750' - -# Holds a list of directories to create on the server. -# -# By default, directories are: -# - created with permissions as specified in `matrix_aux_directory_default_mode` -# - owned by the `matrix_user_username` user and `matrix_user_groupname` group (usually `matrix:matrix`) -# -# Example: -# -# matrix_aux_directory_definitions: -# - dest: /matrix/aux -# -# - dest: /matrix/another -# mode: '0700' -# owner: 'some-user' -# group: 'some-group' -matrix_aux_directory_definitions: [] - -# The default permission mode when creating directories using `matrix_aux_directory_definitions` -matrix_aux_file_default_mode: '0640' - -# Holds a list of files to create on the server. -# -# By default, files are: -# - created with permissions as specified in `matrix_aux_file_default_mode` -# - owned by the `matrix_user_username` user and `matrix_user_groupname` group (usually `matrix:matrix`) -# -# You can define the file content inline (in your `vars.yml` file) or as an external file (see the example below). -# Defining the content inline in `vars.yml` has the benefit of not splitting your configuration into multiple files, -# but rather keeping everything inside `vars.yml` (which also gets backed up on the server in `/matrix/vars.yml`). -# -# Note: parent paths for files must exist. -# If you've defined a file with a destination of `/matrix/some/path/file.txt`, -# then you likely need to add `/matrix/some/path` to `matrix_aux_directory_definitions` as well. -# You don't need to do this for directories that the playbook already creates for you. -# -# Use a `content` key for text content and `src` with a location to a file for binary content. -# The `content` key does not support binary content (see https://github.com/ansible/ansible/issues/11594). -# -# Example: -# -# matrix_aux_file_definitions: -# - dest: "{{ matrix_synapse_config_dir_path }}/something.html" -# content: | -# -# Something -# -# - dest: /matrix/aux/some-other-file.txt -# content: "Something" -# mode: '0600' -# owner: 'some-user' -# group: 'some-group' -# -# - dest: /matrix/aux/yet-another-file.txt -# content: "{{ lookup('template', '/path/to/file.txt.j2') }}" -# mode: '0600' -# owner: 'some-user' -# group: 'some-group' -# -# - dest: /matrix/aux/binary-file.dat -# src: "/path/to/binary.dat" -# mode: '0600' -# owner: 'some-user' -# group: 'some-group' -matrix_aux_file_definitions: [] diff --git a/roles/custom/matrix-aux/tasks/main.yml b/roles/custom/matrix-aux/tasks/main.yml deleted file mode 100644 index 3ae8d179..00000000 --- a/roles/custom/matrix-aux/tasks/main.yml +++ /dev/null @@ -1,9 +0,0 @@ ---- - -- tags: - - setup-all - - setup-aux-files - - install-all - - install-aux-files - block: - - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup.yml" diff --git a/roles/custom/matrix-aux/tasks/setup.yml b/roles/custom/matrix-aux/tasks/setup.yml deleted file mode 100644 index eb0adad4..00000000 --- a/roles/custom/matrix-aux/tasks/setup.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- name: Ensure AUX directories are created - ansible.builtin.file: - dest: "{{ item.dest }}" - state: directory - owner: "{{ item.owner | default(matrix_user_username) }}" - group: "{{ item.group | default(matrix_user_groupname) }}" - mode: "{{ item.mode | default(matrix_aux_directory_default_mode) }}" - with_items: "{{ matrix_aux_directory_definitions }}" - -- name: Ensure AUX files are created - ansible.builtin.copy: - src: "{{ item.src if 'src' in item else omit }}" - content: "{{ item.content if 'content' in item else omit }}" - dest: "{{ item.dest }}" - owner: "{{ item.owner | default(matrix_user_username) }}" - group: "{{ item.group | default(matrix_user_groupname) }}" - mode: "{{ item.mode | default(matrix_aux_file_default_mode) }}" - with_items: "{{ matrix_aux_file_definitions }}" diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml index dbc3b3b8..ee051193 100644 --- a/roles/custom/matrix_playbook_migration/defaults/main.yml +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -20,3 +20,6 @@ matrix_playbook_migration_matrix_redis_migration_validation_enabled: true # Controls if (`matrix_etherpad` -> `etherpad`) validation will run. matrix_playbook_migration_matrix_etherpad_migration_validation_enabled: true + +# Controls if (`matrix_aux` -> `aux`) validation will run. +matrix_playbook_migration_matrix_aux_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index d8fe1683..87292cdb 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -157,3 +157,19 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_etherpad_` -> `etherpad_`). We found usage of the following variables: {{ matrix_playbook_migration_etherpad_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_etherpad_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_aux_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_aux_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_aux_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_aux variables + ansible.builtin.fail: + msg: >- + The matrix-aux role that used to be part of this playbook has been replaced by https://github.com/mother-of-all-self-hosting/ansible-role-aux. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_aux_` -> `aux_`). + We found usage of the following variables: {{ matrix_playbook_migration_aux_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_aux_migration_vars | length > 0" + From d974c0c166012731c5e4ad675c7701c971e77246 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 11:09:59 +0200 Subject: [PATCH 354/382] Make yamllint happy --- roles/custom/matrix_playbook_migration/tasks/validate_config.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 87292cdb..9c958505 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -172,4 +172,3 @@ Please change your configuration (vars.yml) to rename all variables (`matrix_aux_` -> `aux_`). We found usage of the following variables: {{ matrix_playbook_migration_aux_migration_vars.keys() | join(', ') }} when: "matrix_playbook_migration_aux_migration_vars | length > 0" - From abf96569e04c3f8025591a5ee6b5a7786f43f004 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 15:05:19 +0200 Subject: [PATCH 355/382] Add extra_args support to install-service justfile command It's now possible to do: `just install-service prometheus -l host.example.com` --- justfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/justfile b/justfile index 3b5c4de3..f50b5217 100644 --- a/justfile +++ b/justfile @@ -15,8 +15,8 @@ lint: install-all *extra_args: (run-tags "install-all,ensure-matrix-users-created,start" extra_args) # Runs installation tasks for a single service -install-service service: - just --justfile {{ justfile() }} run --tags=install-{{ service }},start-group --extra-vars=group={{ service }} +install-service service *extra_args: + just --justfile {{ justfile() }} run --tags=install-{{ service }},start-group --extra-vars=group={{ service }} {{ extra_args }} # Runs the playbook with --tags=setup-all,ensure-matrix-users-created,start and optional arguments setup-all *extra_args: (run-tags "setup-all,ensure-matrix-users-created,start" extra_args) From 1b6a85e485a73bf07d5b0b34021b98c039165e5c Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 15:09:04 +0200 Subject: [PATCH 356/382] Do not consider prometheus-exporters as part of the prometheus group This makes us rebuild/restart exporters when running `just install-service prometheus`, which we don't like. --- group_vars/matrix_servers | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index bdc2d123..c7750514 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -340,11 +340,11 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-prometheus.service', 'priority': 4000, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-core']}] if matrix_prometheus_enabled else []) + - ([{'name': (prometheus_node_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if prometheus_node_exporter_enabled else []) + ([{'name': (prometheus_node_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus-exporters', 'prometheus-node-exporter']}] if prometheus_node_exporter_enabled else []) + - ([{'name': (prometheus_postgres_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if prometheus_postgres_exporter_enabled else []) + ([{'name': (prometheus_postgres_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus-exporters', 'prometheus-postgres-exporter']}] if prometheus_postgres_exporter_enabled else []) + - ([{'name': 'matrix-prometheus-nginxlog-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if matrix_prometheus_nginxlog_exporter_enabled else []) + ([{'name': 'matrix-prometheus-nginxlog-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus-exporters', 'prometheus-nginxlog-exporter']}] if matrix_prometheus_nginxlog_exporter_enabled else []) + ([{'name': (redis_identifier + '.service'), 'priority': 750, 'groups': ['matrix', 'redis']}] if redis_enabled else []) + From ade312e6f621661e1a6a6ca09ee9c3e1114ee084 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 17:03:19 +0200 Subject: [PATCH 357/382] Add matrix_hookshot_container_network --- roles/custom/matrix-bridge-hookshot/defaults/main.yml | 2 ++ roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml | 5 +++++ .../templates/systemd/matrix-hookshot.service.j2 | 4 ++-- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/roles/custom/matrix-bridge-hookshot/defaults/main.yml b/roles/custom/matrix-bridge-hookshot/defaults/main.yml index db71da79..2113f53a 100644 --- a/roles/custom/matrix-bridge-hookshot/defaults/main.yml +++ b/roles/custom/matrix-bridge-hookshot/defaults/main.yml @@ -200,6 +200,8 @@ matrix_hookshot_permissions: [] matrix_hookshot_bot_displayname: Hookshot Bot matrix_hookshot_bot_avatar: 'mxc://half-shot.uk/2876e89ccade4cb615e210c458e2a7a6883fe17d' +matrix_hookshot_container_network: "{{ matrix_docker_network }}" + # A list of extra arguments to pass to the container matrix_hookshot_container_extra_arguments: [] diff --git a/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml b/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml index f0e8c1c6..576eb007 100644 --- a/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml +++ b/roles/custom/matrix-bridge-hookshot/tasks/setup_install.yml @@ -107,6 +107,11 @@ group: "{{ matrix_user_groupname }}" when: matrix_hookshot_github_enabled | bool and matrix_hookshot_github_private_key|length > 0 +- name: Ensure matrix-hookshot container network is created + community.general.docker_network: + name: "{{ matrix_hookshot_container_network }}" + driver: bridge + - name: Ensure matrix-hookshot.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-hookshot.service.j2" diff --git a/roles/custom/matrix-bridge-hookshot/templates/systemd/matrix-hookshot.service.j2 b/roles/custom/matrix-bridge-hookshot/templates/systemd/matrix-hookshot.service.j2 index 7ebd08b6..713c7a9f 100644 --- a/roles/custom/matrix-bridge-hookshot/templates/systemd/matrix-hookshot.service.j2 +++ b/roles/custom/matrix-bridge-hookshot/templates/systemd/matrix-hookshot.service.j2 @@ -20,8 +20,8 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ - --network={{ matrix_docker_network }} \ - -v {{ matrix_hookshot_base_path }}:/data:z \ + --network={{ matrix_hookshot_container_network }} \ + --mount type=bind,src={{ matrix_hookshot_base_path }},dst=/data \ {% for port in matrix_hookshot_container_http_host_bind_ports %} -p {{ port }} \ {% endfor %} From 21c2bbb11fbd51ffd3cc14e66f5d64fded79e07e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 17:04:23 +0200 Subject: [PATCH 358/382] Add matrix_prometheus_nginxlog_exporter_container_network --- .../matrix-prometheus-nginxlog-exporter/defaults/main.yml | 2 ++ .../tasks/setup_install.yml | 5 +++++ .../systemd/matrix-prometheus-nginxlog-exporter.service.j2 | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/defaults/main.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/defaults/main.yml index 2e85d629..806c751c 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/defaults/main.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/defaults/main.yml @@ -28,6 +28,8 @@ matrix_prometheus_nginxlog_exporter_dashboard_urls: [] matrix_prometheus_nginxlog_exporter_base_path: "{{ matrix_base_data_path }}/prometheus-nginxlog-exporter" matrix_prometheus_nginxlog_exporter_config_path: "{{ matrix_prometheus_nginxlog_exporter_base_path }}/config" +matrix_prometheus_nginxlog_exporter_container_network: "{{ matrix_docker_network }}" + # A list of extra docker arguments to pass to the container matrix_prometheus_nginxlog_exporter_container_extra_arguments: [] diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml index ca3bf3a8..8d3fe1d4 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml @@ -30,6 +30,11 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" +- name: Ensure prometheus-nginxlog-exporter container network is created + community.general.docker_network: + name: "{{ matrix_prometheus_nginxlog_exporter_container_network }}" + driver: bridge + - name: Ensure matrix-prometheus-nginxlog-exporter.service installed ansible.builtin.template: src: "{{ role_path }}/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2" diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2 b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2 index e799380e..0d01aa21 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2 +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2 @@ -22,7 +22,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_prometheus_nginxlog_exporter_container_network }} \ {% if matrix_prometheus_nginxlog_exporter_container_http_host_bind_port %} -p {{ matrix_prometheus_nginxlog_exporter_container_http_host_bind_port }}:{{ matrix_prometheus_nginxlog_exporter_container_metrics_port }} \ {% endif %} From a542a2ff41f1a87ea6c6efa60528ee060c9f63ef Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 17:59:08 +0200 Subject: [PATCH 359/382] Upgrade devture/ansible (2.13.6-r0-2 -> 2.13.6-r0-3) --- docs/ansible.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ansible.md b/docs/ansible.md index dcda338a..6b246d54 100644 --- a/docs/ansible.md +++ b/docs/ansible.md @@ -65,7 +65,7 @@ docker run -it --rm \ -w /work \ -v `pwd`:/work \ --entrypoint=/bin/sh \ -docker.io/devture/ansible:2.13.6-r0-2 +docker.io/devture/ansible:2.13.6-r0-3 ``` Once you execute the above command, you'll be dropped into a `/work` directory inside a Docker container. From a1a37700b916102b757d9e76c0cdb2500a2ec43e Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 18:40:01 +0200 Subject: [PATCH 360/382] Update one more devture/ansible reference --- docs/ansible.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ansible.md b/docs/ansible.md index 6b246d54..6018860e 100644 --- a/docs/ansible.md +++ b/docs/ansible.md @@ -86,7 +86,7 @@ docker run -it --rm \ -v `pwd`:/work \ -v $HOME/.ssh/id_rsa:/root/.ssh/id_rsa:ro \ --entrypoint=/bin/sh \ -docker.io/devture/ansible:2.13.6-r0 +docker.io/devture/ansible:2.13.6-r0-3 ``` The above command tries to mount an SSH key (`$HOME/.ssh/id_rsa`) into the container (at `/root/.ssh/id_rsa`). From 01db1ae9b70c2421a8a94fe5c06b324cb1d99c99 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Mon, 20 Mar 2023 18:44:01 +0200 Subject: [PATCH 361/382] Mention passlib It's not strictly required yet, but certain versions of Ansible display warnings if passlib is missing. The non-passlib crypto usage is deprecated, so passlib will become a requirement in newer Ansible versions. It's only a matter of time. --- docs/prerequisites.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/prerequisites.md b/docs/prerequisites.md index 83ae42da..be60a5f4 100644 --- a/docs/prerequisites.md +++ b/docs/prerequisites.md @@ -20,6 +20,8 @@ If your distro runs within an [LXC container](https://linuxcontainers.org/), you - The [Ansible](http://ansible.com/) program being installed on your own computer. It's used to run this playbook and configures your server for you. Take a look at [our guide about Ansible](ansible.md) for more information, as well as [version requirements](ansible.md#supported-ansible-versions) and alternative ways to run Ansible. +- the [passlib](https://passlib.readthedocs.io/en/stable/index.html) Python library installed on the computer you run Ansible. On most distros, you need to install some `python-passlib` or `py3-passlib` package, etc. + - [`git`](https://git-scm.com/) is the recommended way to download the playbook to your computer. `git` may also be required on the server if you will be [self-building](self-building.md) components. - [`just`](https://github.com/casey/just) for running `just roles`, etc. (see [`justfile`](../justfile)), although you can also run these commands manually From d351213486400af1cc794a5b8d0e7604dcb115be Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 21 Mar 2023 07:29:21 +0200 Subject: [PATCH 362/382] Upgrade prometheus_node_exporter (v1.5.0-4 -> v1.5.0-6) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index ffcd86af..c1514016 100644 --- a/requirements.yml +++ b/requirements.yml @@ -31,7 +31,7 @@ version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git - version: v1.5.0-4 + version: v1.5.0-6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.11.1-2 From 14b8efcad262d49958c30df904bcef4dc8153624 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 21 Mar 2023 07:38:12 +0200 Subject: [PATCH 363/382] Replace matrix-prometheus with an external Prometheus role --- CHANGELOG.md | 21 ++- ...configuring-playbook-prometheus-grafana.md | 6 +- ...onfiguring-playbook-prometheus-nginxlog.md | 4 +- group_vars/matrix_servers | 117 ++++++++++---- playbooks/matrix.yml | 3 +- requirements.yml | 4 + .../tasks/setup_uninstall.yml | 2 +- .../tasks/validate_config.yml | 4 +- .../defaults/main.yml | 144 ++++++++++++++++++ .../tasks/install_synapse_rules.yml | 14 ++ .../tasks/main.yml | 11 ++ .../matrix-prometheus/defaults/main.yml | 90 ----------- roles/custom/matrix-prometheus/tasks/main.yml | 20 --- .../matrix-prometheus/tasks/setup_install.yml | 57 ------- .../tasks/setup_uninstall.yml | 20 --- .../tasks/validate_config.yml | 15 -- .../templates/prometheus.yml.j2 | 73 --------- .../systemd/matrix-prometheus.service.j2 | 51 ------- .../tasks/validate_config.yml | 19 +++ 19 files changed, 311 insertions(+), 364 deletions(-) create mode 100644 roles/custom/matrix-prometheus-services-connect/defaults/main.yml create mode 100644 roles/custom/matrix-prometheus-services-connect/tasks/install_synapse_rules.yml create mode 100644 roles/custom/matrix-prometheus-services-connect/tasks/main.yml delete mode 100644 roles/custom/matrix-prometheus/defaults/main.yml delete mode 100644 roles/custom/matrix-prometheus/tasks/main.yml delete mode 100644 roles/custom/matrix-prometheus/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-prometheus/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-prometheus/tasks/validate_config.yml delete mode 100644 roles/custom/matrix-prometheus/templates/prometheus.yml.j2 delete mode 100644 roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ba17ea3..0c9de0ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,22 @@ +# 2023-03-21 + +## The matrix-prometheus role lives independently now + +**TLDR**: the `matrix-prometheus` role is now included from the [ansible-role-prometheus](https://github.com/mother-of-all-self-hosting/ansible-role-prometheus) repository, part of the [MASH playbook](https://github.com/mother-of-all-self-hosting/mash-playbook). Some variables have been renamed. All functionality remains intact. + +The `matrix-prometheus` role has been relocated in its own repository, part of the [MASH playbook](https://github.com/mother-of-all-self-hosting/mash-playbook) project - an Ansible playbook for self-hosting [a growing list of FOSS software](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/docs/supported-services.md). If hosting a Prometheus stack on the Matrix server itself did not stand right with you or you always wanted to host most stuff, you can now use this new playbook to do so. + +Extracting the Prometheus role out of this Matrix playbook required huge internal refactoring to the way the Prometheus configuration (scraping jobs) is generated. If you notice any breakage after upgrading, let us know. + +You need to **update you roles** (`just roles` or `make roles`) regardless of whether you're using Prometheus or not. + +If you're making use of Prometheus via this playbook, you will need to update variable references in your `vars.yml` file: + + - `matrix_prometheus_docker_image_` -> `matrix_prometheus_container_image_` + - `matrix_prometheus_` -> `prometheus_` + - some other internal variables have changed, but the playbook will tell you about them + + # 2023-03-12 ## synapse-auto-compressor support @@ -830,7 +849,7 @@ See our [Setting up the ntfy push notifications server](docs/configuring-playboo **If you are using the [Hookshot bridge](docs/configuring-playbook-bridge-hookshot.md)**, you may find that: 1. **Metrics may not be enabled by default anymore**: - - If Prometheus is enabled (`matrix_prometheus_enabled: true`), then Hookshot metrics will be enabled automatically (`matrix_hookshot_metrics_enabled: true`). These metrics will be collected from the local (in-container) Prometheus over the container network. + - If Prometheus is enabled (`prometheus_enabled: true`), then Hookshot metrics will be enabled automatically (`matrix_hookshot_metrics_enabled: true`). These metrics will be collected from the local (in-container) Prometheus over the container network. - **If Prometheus is not enabled** (you are either not using Prometheus or are using an external one), **Hookshot metrics will not be enabled by default anymore**. Feel free to enable them by setting `matrix_hookshot_metrics_enabled: true`. Also, see below. 2. When metrics are meant to be **consumed by an external Prometheus server**, `matrix_hookshot_metrics_proxying_enabled` needs to be set to `true`, so that metrics would be exposed (proxied) "publicly" on `https://matrix.DOMAIN/metrics/hookshot`. To make use of this, you'll also need to enable the new `https://matrix.DOMAIN/metrics/*` endpoints mentioned above, using `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. Learn more in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. 3. **We've changed the URL we're exposing Hookshot metrics at** for external Prometheus servers. Until now, you were advised to consume Hookshot metrics from `https://stats.DOMAIN/hookshot/metrics` (working in conjunction with `matrix_nginx_proxy_proxy_synapse_metrics`). From now on, **this no longer works**. As described above, you need to start consuming metrics from `https://matrix.DOMAIN/metrics/hookshot`. diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index de05d690..7e4764c2 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -7,7 +7,7 @@ You can enable this with the following settings in your configuration file (`inv Remember to add `stats.` to DNS as described in [Configuring DNS](configuring-dns.md) before running the playbook. ```yaml -matrix_prometheus_enabled: true +prometheus_enabled: true # You can remove this, if unnecessary. prometheus_node_exporter_enabled: true @@ -39,7 +39,7 @@ The retention policy of Prometheus metrics is [15 days by default](https://prome Name | Description -----|---------- -`matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. +`prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. `prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures `prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. `matrix_prometheus_nginxlog_exporter_enabled`|[NGINX Log Exporter](configuring-playbook-prometheus-nginxlog.md) is an addon of sorts to expose NGINX logs to Prometheus. @@ -57,7 +57,7 @@ Most of our docker containers run with limited system access, but the `prometheu ## Collecting metrics to an external Prometheus server -**If the integrated Prometheus server is enabled** (`matrix_prometheus_enabled: true`), metrics are collected by it from each service via communication that happens over the container network. Each service does not need to expose its metrics "publicly". +**If the integrated Prometheus server is enabled** (`prometheus_enabled: true`), metrics are collected by it from each service via communication that happens over the container network. Each service does not need to expose its metrics "publicly". When you'd like **to collect metrics from an external Prometheus server**, you need to expose service metrics outside of the container network. diff --git a/docs/configuring-playbook-prometheus-nginxlog.md b/docs/configuring-playbook-prometheus-nginxlog.md index 321a1d6e..55272de4 100644 --- a/docs/configuring-playbook-prometheus-nginxlog.md +++ b/docs/configuring-playbook-prometheus-nginxlog.md @@ -12,7 +12,7 @@ You can enable this role by adding the following settings in your configuration matrix_prometheus_nginxlog_exporter_enabled: true # required depency -matrix_prometheus_enabled: true +prometheus_enabled: true # optional for visualization grafana_enabled: true @@ -20,7 +20,7 @@ grafana_enabled: true x | Prerequisites | Variable | Description |:--:|:--:|:--:|:--| -**REQUIRED** | `matrix-prometheus`| `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. +**REQUIRED** | `matrix-prometheus`| `prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. _Optional_ | [`matrix-grafana`](configuring-playbook-prometheus-grafana.md) | [`grafana_enabled`](configuring-playbook-prometheus-grafana.md)|[Grafana](https://grafana.com) is the visual component. It shows (on the `stats.` subdomain) graphs that we're interested in. When enabled the `NGINX PROXY` dashboard is automatically added. ## Docker Image Compatibility diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index c7750514..0b8d7e25 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -338,7 +338,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': (devture_postgres_backup_identifier + '.service'), 'priority': 5000, 'groups': ['matrix', 'backup', 'postgres-backup']}] if devture_postgres_backup_enabled else []) + - ([{'name': 'matrix-prometheus.service', 'priority': 4000, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-core']}] if matrix_prometheus_enabled else []) + ([{'name': (prometheus_identifier + '.service'), 'priority': 4000, 'groups': ['matrix', 'monitoring', 'prometheus']}] if prometheus_enabled else []) + ([{'name': (prometheus_node_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus-exporters', 'prometheus-node-exporter']}] if prometheus_node_exporter_enabled else []) + @@ -871,7 +871,7 @@ matrix_mautrix_facebook_bridge_presence: "{{ matrix_synapse_presence_enabled if # - `matrix_mautrix_facebook_metrics_enabled` # - `matrix_mautrix_facebook_proxying_metrics_enabled` # - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` -matrix_mautrix_facebook_metrics_enabled: "{{ matrix_prometheus_enabled }}" +matrix_mautrix_facebook_metrics_enabled: "{{ prometheus_enabled }}" # We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain # and point them to a migration path. @@ -1004,7 +1004,7 @@ matrix_mautrix_instagram_bridge_presence: "{{ matrix_synapse_presence_enabled if # - `matrix_mautrix_instagram_metrics_enabled` # - `matrix_mautrix_instagram_proxying_metrics_enabled` # - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` -matrix_mautrix_instagram_metrics_enabled: "{{ matrix_prometheus_enabled }}" +matrix_mautrix_instagram_metrics_enabled: "{{ prometheus_enabled }}" # We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain # and point them to a migration path. @@ -1055,7 +1055,7 @@ matrix_mautrix_signal_login_shared_secret: "{{ matrix_synapse_ext_password_provi # - `matrix_mautrix_signal_metrics_enabled` # - `matrix_mautrix_signal_proxying_metrics_enabled` # - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` -matrix_mautrix_signal_metrics_enabled: "{{ matrix_prometheus_enabled }}" +matrix_mautrix_signal_metrics_enabled: "{{ prometheus_enabled }}" matrix_mautrix_signal_database_engine: 'postgres' matrix_mautrix_signal_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" @@ -1185,7 +1185,7 @@ matrix_mautrix_whatsapp_login_shared_secret: "{{ matrix_synapse_ext_password_pro # - `matrix_mautrix_twitter_metrics_enabled` # - `matrix_mautrix_twitter_proxying_metrics_enabled` # - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` -matrix_mautrix_twitter_metrics_enabled: "{{ matrix_prometheus_enabled }}" +matrix_mautrix_twitter_metrics_enabled: "{{ prometheus_enabled }}" # Postgres is the default, except if not using internal Postgres server matrix_mautrix_whatsapp_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -1291,7 +1291,7 @@ matrix_hookshot_provisioning_enabled: "{{ matrix_hookshot_provisioning_secret an # - `matrix_hookshot_metrics_enabled` # - `matrix_hookshot_metrics_proxying_enabled` # - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` -matrix_hookshot_metrics_enabled: "{{ matrix_prometheus_enabled }}" +matrix_hookshot_metrics_enabled: "{{ prometheus_enabled }}" matrix_hookshot_urlprefix_port_enabled: "{{ matrix_nginx_proxy_container_https_host_bind_port == 443 if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_https_host_bind_port == 80 }}" matrix_hookshot_urlprefix_port: ":{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}" @@ -3034,7 +3034,7 @@ devture_postgres_backup_databases: "{{ devture_postgres_managed_databases | map( matrix_sygnal_enabled: false # If someone instals Prometheus via the playbook, they most likely wish to monitor Sygnal. -matrix_sygnal_metrics_prometheus_enabled: "{{ matrix_prometheus_enabled }}" +matrix_sygnal_metrics_prometheus_enabled: "{{ prometheus_enabled }}" matrix_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}" @@ -3287,7 +3287,7 @@ matrix_synapse_tls_private_key_path: ~ matrix_synapse_federation_port_openid_resource_required: "{{ not matrix_synapse_federation_enabled and (matrix_dimension_enabled or matrix_ma1sd_enabled or matrix_user_verification_service_enabled) }}" # If someone instals Prometheus via the playbook, they most likely wish to monitor Synapse. -matrix_synapse_metrics_enabled: "{{ matrix_prometheus_enabled }}" +matrix_synapse_metrics_enabled: "{{ prometheus_enabled }}" matrix_synapse_email_enabled: "{{ matrix_mailer_enabled }}" matrix_synapse_email_smtp_host: "matrix-mailer" @@ -3452,6 +3452,7 @@ matrix_synapse_admin_container_labels_traefik_tls_certResolver: "{{ devture_trae # ###################################################################### + ###################################################################### # # etke/prometheus_node_exporter @@ -3486,6 +3487,7 @@ prometheus_node_exporter_container_labels_traefik_tls_certResolver: "{{ devture_ # ###################################################################### + ###################################################################### # # etke/prometheus_postgres_exporter @@ -3530,6 +3532,7 @@ prometheus_postgres_exporter_systemd_required_services_list: | # ###################################################################### + ###################################################################### # # matrix-prometheus-nginxlog-exporter @@ -3544,45 +3547,101 @@ matrix_prometheus_nginxlog_exporter_enabled: false # ###################################################################### + ###################################################################### # -# matrix-prometheus +# prometheus # ###################################################################### -matrix_prometheus_enabled: false +prometheus_enabled: false + +prometheus_identifier: matrix-prometheus -matrix_prometheus_container_network: "{{ matrix_docker_network }}" +prometheus_base_path: "{{ matrix_base_data_path }}/prometheus" + +prometheus_uid: "{{ matrix_user_uid }}" +prometheus_gid: "{{ matrix_user_gid }}" + +prometheus_container_additional_networks: | + {{ + ( + ([prometheus_node_exporter_container_network] if prometheus_self_node_scraper_enabled and prometheus_node_exporter_container_network != prometheus_container_network else []) + + + ([matrix_synapse_container_network] if matrix_prometheus_services_connect_scraper_synapse_enabled and matrix_synapse_container_network != prometheus_container_network else []) + + + ([prometheus_postgres_exporter_container_network] if matrix_prometheus_services_connect_scraper_postgres_enabled and prometheus_postgres_exporter_container_network != prometheus_container_network else []) + + + ([matrix_hookshot_container_network] if matrix_prometheus_services_connect_scraper_hookshot_enabled and matrix_hookshot_container_network != prometheus_container_network else []) + + + ([matrix_prometheus_nginxlog_exporter_container_network] if matrix_prometheus_services_connect_scraper_nginxlog_enabled and matrix_prometheus_nginxlog_exporter_container_network != prometheus_container_network else []) + ) | unique + }} # Normally, matrix-nginx-proxy is enabled and nginx can reach Prometheus over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # Prometheus' HTTP port to the local host. -matrix_prometheus_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9090') if matrix_playbook_service_host_bind_interface_prefix else '' }}" +prometheus_container_http_host_bind_port: "{{ (matrix_playbook_service_host_bind_interface_prefix ~ '9090') if matrix_playbook_service_host_bind_interface_prefix else '' }}" + +prometheus_config_rule_files_auto: | + {{ + (['synapse-v2.rules'] if matrix_prometheus_services_connect_synapse_rules_enabled else []) + }} + +prometheus_self_node_scraper_enabled: "{{ prometheus_node_exporter_enabled }}" +prometheus_self_node_scraper_static_configs_target: "{{ (prometheus_node_exporter_identifier + ':9100') if prometheus_node_exporter_enabled else '' }}" + +prometheus_config_scrape_configs_auto: | + {{ + (matrix_prometheus_services_connect_scraper_synapse_configs if matrix_prometheus_services_connect_scraper_synapse_enabled else []) + + + (matrix_prometheus_services_connect_scraper_postgres_scrape_configs if matrix_prometheus_services_connect_scraper_postgres_enabled else []) + + + (matrix_prometheus_services_connect_scraper_hookshot_scrape_configs if matrix_prometheus_services_connect_scraper_hookshot_enabled else []) + + + (matrix_prometheus_services_connect_scraper_nginxlog_scrape_configs if matrix_prometheus_services_connect_scraper_nginxlog_enabled else []) + }} + +###################################################################### +# +# /prometheus +# +###################################################################### -matrix_prometheus_scraper_synapse_enabled: "{{ matrix_synapse_enabled and matrix_synapse_metrics_enabled }}" -matrix_prometheus_scraper_synapse_targets: ['matrix-synapse:{{ matrix_synapse_metrics_port }}'] -matrix_prometheus_scraper_synapse_workers_enabled_list: "{{ matrix_synapse_workers_enabled_list }}" -matrix_prometheus_scraper_synapse_rules_synapse_tag: "{{ matrix_synapse_docker_image_tag }}" -matrix_prometheus_scraper_node_enabled: "{{ prometheus_node_exporter_enabled }}" -matrix_prometheus_scraper_node_targets: "{{ [prometheus_node_exporter_identifier + ':9100'] if prometheus_node_exporter_enabled else [] }}" +###################################################################### +# +# matrix-prometheus-services-connect +# +###################################################################### + +matrix_prometheus_services_connect_synapse_enabled: "{{ prometheus_enabled and matrix_synapse_enabled }}" -matrix_prometheus_scraper_postgres_enabled: "{{ prometheus_postgres_exporter_enabled }}" -matrix_prometheus_scraper_postgres_targets: "{{ [prometheus_postgres_exporter_identifier + ':' + prometheus_postgres_exporter_port | string] if matrix_prometheus_scraper_postgres_enabled else [] }}" +matrix_prometheus_services_connect_synapse_rules_download_synapse_tag: "{{ matrix_synapse_docker_image_tag }}" +matrix_prometheus_services_connect_synapse_rules_download_dir_path: "{{ prometheus_config_path }}" +matrix_prometheus_services_connect_synapse_rules_download_owner: "{{ prometheus_uid }}" +matrix_prometheus_services_connect_synapse_rules_download_group: "{{ prometheus_gid }}" -matrix_prometheus_scraper_hookshot_enabled: "{{ matrix_hookshot_metrics_enabled|default(false) }}" -matrix_prometheus_scraper_hookshot_targets: "{{ [matrix_hookshot_container_url | string +':'+ matrix_hookshot_metrics_port | string] if matrix_hookshot_metrics_enabled else [] }}" +matrix_prometheus_services_connect_scraper_synapse_enabled: "{{ matrix_synapse_enabled and matrix_synapse_metrics_enabled }}" +matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs_target: "matrix-synapse:{{ matrix_synapse_metrics_port }}" +matrix_prometheus_services_connect_scraper_synapse_workers_enabled_list: "{{ matrix_synapse_workers_enabled_list }}" -matrix_prometheus_scraper_nginxlog_enabled: "{{ matrix_prometheus_nginxlog_exporter_enabled }}" -matrix_prometheus_scraper_nginxlog_server_port: "{{ (matrix_prometheus_nginxlog_exporter_container_hostname | string +':'+ matrix_prometheus_nginxlog_exporter_container_metrics_port | string) -| default('') }}" +matrix_prometheus_services_connect_scraper_postgres_enabled: "{{ prometheus_postgres_exporter_enabled }}" +matrix_prometheus_services_connect_scraper_postgres_static_configs_target: "{{ prometheus_postgres_exporter_identifier }}:{{ prometheus_postgres_exporter_port | string }}" + +matrix_prometheus_services_connect_scraper_hookshot_enabled: "{{ matrix_hookshot_enabled and matrix_hookshot_metrics_enabled }}" +matrix_prometheus_services_connect_scraper_hookshot_static_configs_target: "{{ matrix_hookshot_container_url }}:{{ matrix_hookshot_metrics_port | string }}" + +matrix_prometheus_services_connect_scraper_nginxlog_enabled: "{{ matrix_prometheus_nginxlog_exporter_enabled }}" +matrix_prometheus_services_connect_scraper_nginxlog_static_configs_target: "{{ matrix_prometheus_nginxlog_exporter_container_hostname }}:{{ matrix_prometheus_nginxlog_exporter_container_metrics_port | string }}" ###################################################################### # -# /matrix-prometheus +# /matrix-prometheus-services-connect # ###################################################################### + ###################################################################### # # etke/grafana @@ -3607,7 +3666,7 @@ grafana_container_additional_networks: | ( ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + - ([matrix_prometheus_container_network] if matrix_prometheus_enabled and matrix_prometheus_container_network != grafana_container_network else []) + ([prometheus_container_network] if prometheus_enabled and prometheus_container_network != grafana_container_network else []) ) | unique }} @@ -3628,7 +3687,7 @@ grafana_provisioning_datasources: | 'type': 'prometheus', 'access': 'proxy', 'url': 'http://matrix-prometheus:9090', - }] if matrix_prometheus_enabled else []) + }] if prometheus_enabled else []) }} grafana_dashboard_download_urls: | @@ -3665,6 +3724,8 @@ grafana_default_home_dashboard_path: |- # ###################################################################### + + ###################################################################### # # matrix-registration diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 93e9208f..5bf19b9d 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -92,8 +92,9 @@ - galaxy/prometheus_node_exporter - galaxy/prometheus_postgres_exporter - custom/matrix-prometheus-nginxlog-exporter - - custom/matrix-prometheus + - galaxy/prometheus - galaxy/grafana + - custom/matrix-prometheus-services-connect - custom/matrix-prometheus-services-proxy-connect - custom/matrix-registration - custom/matrix-client-element diff --git a/requirements.yml b/requirements.yml index c1514016..ebd3fd0f 100644 --- a/requirements.yml +++ b/requirements.yml @@ -63,3 +63,7 @@ - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-aux.git name: aux version: v1.0.0-0 + +- src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus.git + name: prometheus + version: v2.42.0-0 diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml index 7d27cbae..93c4aefc 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml @@ -13,7 +13,7 @@ This role has added to configs in 'matrix-nginx-proxy', 'matrix-synapse-reverse-proxy-companion' and 'matrix-prometheus'. Running 'setup-synapse-reverse-proxy-companion' WILL NOT remove those settings from those roles. Run the playbook again with the `setup-all` tag or all three 'setup-nginx-proxy,setup-synapse-reverse-proxy-companion,setup-prometheus' tags while - 'matrix_prometheus_enabled: false' to rebuild their configs. + 'prometheus_enabled: false' to rebuild their configs. when: not ('setup-all' in ansible_run_tags or ('setup-nginx-proxy' in ansible_run_tags and 'setup-synapse-reverse-proxy-companion' in ansible_run_tags and 'setup-prometheus' in ansible_run_tags)) - name: Ensure matrix-prometheus-nginxlog-exporter is stopped diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/validate_config.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/validate_config.yml index a8eccaee..2e308736 100644 --- a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/validate_config.yml +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/validate_config.yml @@ -3,9 +3,9 @@ - name: Fail if Prometheus not enabled ansible.builtin.fail: msg: > - You need to set `matrix_prometheus_enabled: true` or configure an external Prometheus database + You need to set `prometheus_enabled: true` or configure an external Prometheus database as this roles requires Prometheus for data storage. - when: not matrix_prometheus_enabled and not matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled + when: not prometheus_enabled and not matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled - name: Fail if docker image not availble for arch ansible.builtin.fail: diff --git a/roles/custom/matrix-prometheus-services-connect/defaults/main.yml b/roles/custom/matrix-prometheus-services-connect/defaults/main.yml new file mode 100644 index 00000000..748f88c5 --- /dev/null +++ b/roles/custom/matrix-prometheus-services-connect/defaults/main.yml @@ -0,0 +1,144 @@ +--- + +# matrix-prometheus-services-connect is a role which helps integrate +# the various Matrix services with the Prometheus role. + +# Controls whether Synapse integration should be done +matrix_prometheus_services_connect_synapse_enabled: false + +# Controls whether the synapse rules shall be loaded into Prometheus +matrix_prometheus_services_connect_synapse_rules_enabled: "{{ matrix_prometheus_services_connect_synapse_enabled }}" + +# Controls whether the synapse rules shall be downloaded +matrix_prometheus_services_connect_synapse_rules_download_enabled: "{{ matrix_prometheus_services_connect_synapse_rules_enabled }}" +matrix_prometheus_services_connect_synapse_rules_download_synapse_tag: master +matrix_prometheus_services_connect_synapse_rules_download_src_url: "https://raw.githubusercontent.com/matrix-org/synapse/{{ matrix_prometheus_services_connect_synapse_rules_download_synapse_tag }}/contrib/prometheus/synapse-v2.rules" +matrix_prometheus_services_connect_synapse_rules_download_dir_path: '' +matrix_prometheus_services_connect_synapse_rules_download_owner: '' +matrix_prometheus_services_connect_synapse_rules_download_group: '' + +# Controls whether Synapse shall be scraped +matrix_prometheus_services_connect_scraper_synapse_enabled: false + +# Controls scraping the Synapse processes +matrix_prometheus_services_connect_scraper_synapse_job_name: synapse +matrix_prometheus_services_connect_scraper_synapse_metrics_path: /_synapse/metrics +matrix_prometheus_services_connect_scraper_synapse_scrape_interval: 15s +matrix_prometheus_services_connect_scraper_synapse_scrape_timeout: 15s + +matrix_prometheus_services_connect_scraper_synapse_configs: | + {{ + [{ + 'job_name': matrix_prometheus_services_connect_scraper_synapse_job_name, + 'metrics_path': matrix_prometheus_services_connect_scraper_synapse_metrics_path, + 'scrape_interval': matrix_prometheus_services_connect_scraper_synapse_scrape_interval, + 'scrape_timeout': matrix_prometheus_services_connect_scraper_synapse_scrape_timeout, + 'static_configs': matrix_prometheus_services_connect_scraper_synapse_static_configs, + }] + }} + +matrix_prometheus_services_connect_scraper_synapse_static_configs: | + {{ + (matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs if matrix_prometheus_services_connect_scraper_synapse_main_process_enabled else []) + + + (matrix_prometheus_services_connect_scraper_synapse_workers_static_configs if matrix_prometheus_services_connect_scraper_synapse_workers_enabled else []) + }} + +# Controls whether the Synapse main process shall be scraped +matrix_prometheus_services_connect_scraper_synapse_main_process_enabled: "{{ matrix_prometheus_services_connect_scraper_synapse_enabled }}" + +matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs: | + {{ + [{ + 'targets': [matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs_target], + 'labels': { + 'instance': matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs_instance, + 'job': matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs_job, + 'index': 0, + } + }] + }} +matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs_target: '' +matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs_instance: "{{ matrix_domain }}" +matrix_prometheus_services_connect_scraper_synapse_main_process_static_configs_job: master + +# Controls whether Synapse workers shall be scraped +matrix_prometheus_services_connect_scraper_synapse_workers_enabled: "{{ matrix_prometheus_services_connect_scraper_synapse_enabled and matrix_prometheus_services_connect_scraper_synapse_workers_enabled_list | length > 0 }}" + +matrix_prometheus_services_connect_scraper_synapse_workers_enabled_list: [] + +matrix_prometheus_services_connect_scraper_synapse_workers_static_configs_as_yaml: | + {% for worker in matrix_prometheus_services_connect_scraper_synapse_workers_enabled_list %} + {% if worker.metrics_port != 0 %} + - targets: ['{{ worker.name }}:{{ worker.metrics_port }}'] + labels: + instance: {{ matrix_prometheus_services_connect_scraper_synapse_workers_static_configs_instance | to_json }} + worker_id: {{ worker.id | to_json }} + job: {{ worker.type | to_json }} + app: {{ worker.app | to_json }} + {% endif %} + {% endfor %} + +matrix_prometheus_services_connect_scraper_synapse_workers_static_configs_instance: "{{ matrix_domain }}" + +matrix_prometheus_services_connect_scraper_synapse_workers_static_configs: "{{ [] if matrix_prometheus_services_connect_scraper_synapse_workers_enabled_list | length == 0 else matrix_prometheus_services_connect_scraper_synapse_workers_static_configs_as_yaml | from_yaml }}" + +# Controls whether Postgres (postgres-exporter) shall be scraped +matrix_prometheus_services_connect_scraper_postgres_enabled: false +matrix_prometheus_services_connect_scraper_postgres_job_name: postgres +matrix_prometheus_services_connect_scraper_postgres_metrics_path: /metrics +matrix_prometheus_services_connect_scraper_postgres_scrape_interval: 15s +matrix_prometheus_services_connect_scraper_postgres_scrape_timeout: 15s +matrix_prometheus_services_connect_scraper_postgres_static_configs: "{{ [{'targets': [matrix_prometheus_services_connect_scraper_postgres_static_configs_target]}] }}" +matrix_prometheus_services_connect_scraper_postgres_static_configs_target: '' +# The final scrape config for the Postgers scraper +matrix_prometheus_services_connect_scraper_postgres_scrape_configs: | + {{ + [{ + 'job_name': matrix_prometheus_services_connect_scraper_postgres_job_name, + 'metrics_path': matrix_prometheus_services_connect_scraper_postgres_metrics_path, + 'scrape_interval': matrix_prometheus_services_connect_scraper_postgres_scrape_interval, + 'scrape_timeout': matrix_prometheus_services_connect_scraper_postgres_scrape_timeout, + 'static_configs': matrix_prometheus_services_connect_scraper_postgres_static_configs, + }] + }} + +# Controls whether Hookshot shall be scraped +matrix_prometheus_services_connect_scraper_hookshot_enabled: false +matrix_prometheus_services_connect_scraper_hookshot_job_name: hookshot +matrix_prometheus_services_connect_scraper_hookshot_metrics_path: /metrics +matrix_prometheus_services_connect_scraper_hookshot_scrape_interval: 15s +matrix_prometheus_services_connect_scraper_hookshot_scrape_timeout: 15s +matrix_prometheus_services_connect_scraper_hookshot_static_configs: "{{ [{'targets': [matrix_prometheus_services_connect_scraper_hookshot_static_configs_target]}] }}" +matrix_prometheus_services_connect_scraper_hookshot_static_configs_target: '' +# The final scrape config for the Hookshot scraper +matrix_prometheus_services_connect_scraper_hookshot_scrape_configs: | + {{ + [{ + 'job_name': matrix_prometheus_services_connect_scraper_hookshot_job_name, + 'metrics_path': matrix_prometheus_services_connect_scraper_hookshot_metrics_path, + 'scrape_interval': matrix_prometheus_services_connect_scraper_hookshot_scrape_interval, + 'scrape_timeout': matrix_prometheus_services_connect_scraper_hookshot_scrape_timeout, + 'static_configs': matrix_prometheus_services_connect_scraper_hookshot_static_configs, + }] + }} + +# Controls whether nginxlog shall be scraped +matrix_prometheus_services_connect_scraper_nginxlog_enabled: false +matrix_prometheus_services_connect_scraper_nginxlog_job_name: nginxlog +matrix_prometheus_services_connect_scraper_nginxlog_metrics_path: /metrics +matrix_prometheus_services_connect_scraper_nginxlog_scrape_interval: 15s +matrix_prometheus_services_connect_scraper_nginxlog_scrape_timeout: 15s +matrix_prometheus_services_connect_scraper_nginxlog_static_configs: "{{ [{'targets': [matrix_prometheus_services_connect_scraper_nginxlog_static_configs_target]}] }}" +matrix_prometheus_services_connect_scraper_nginxlog_static_configs_target: '' +# The final scrape config for the nginxlog scraper +matrix_prometheus_services_connect_scraper_nginxlog_scrape_configs: | + {{ + [{ + 'job_name': matrix_prometheus_services_connect_scraper_nginxlog_job_name, + 'metrics_path': matrix_prometheus_services_connect_scraper_nginxlog_metrics_path, + 'scrape_interval': matrix_prometheus_services_connect_scraper_nginxlog_scrape_interval, + 'scrape_timeout': matrix_prometheus_services_connect_scraper_nginxlog_scrape_timeout, + 'static_configs': matrix_prometheus_services_connect_scraper_nginxlog_static_configs, + }] + }} diff --git a/roles/custom/matrix-prometheus-services-connect/tasks/install_synapse_rules.yml b/roles/custom/matrix-prometheus-services-connect/tasks/install_synapse_rules.yml new file mode 100644 index 00000000..78b218ac --- /dev/null +++ b/roles/custom/matrix-prometheus-services-connect/tasks/install_synapse_rules.yml @@ -0,0 +1,14 @@ +--- + +- name: Download synapse-v2.rules + ansible.builtin.get_url: + url: "{{ matrix_prometheus_services_connect_synapse_rules_download_src_url }}" + dest: "{{ matrix_prometheus_services_connect_synapse_rules_download_dir_path }}/synapse-v2.rules" + force: true + mode: 0440 + owner: "{{ matrix_prometheus_services_connect_synapse_rules_download_owner }}" + group: "{{ matrix_prometheus_services_connect_synapse_rules_download_group }}" + register: result + retries: "{{ devture_playbook_help_geturl_retries_count }}" + delay: "{{ devture_playbook_help_geturl_retries_delay }}" + until: result is not failed diff --git a/roles/custom/matrix-prometheus-services-connect/tasks/main.yml b/roles/custom/matrix-prometheus-services-connect/tasks/main.yml new file mode 100644 index 00000000..f2c22b7f --- /dev/null +++ b/roles/custom/matrix-prometheus-services-connect/tasks/main.yml @@ -0,0 +1,11 @@ +--- + +- tags: + - setup-all + - setup-prometheus + - setup-prometheus-services-connect + - install-all + - install-prometheus-services-connect + block: + - when: matrix_prometheus_services_connect_synapse_rules_download_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/install_synapse_rules.yml" diff --git a/roles/custom/matrix-prometheus/defaults/main.yml b/roles/custom/matrix-prometheus/defaults/main.yml deleted file mode 100644 index fddacfb4..00000000 --- a/roles/custom/matrix-prometheus/defaults/main.yml +++ /dev/null @@ -1,90 +0,0 @@ ---- -# matrix-prometheus is an open-source systems monitoring and alerting toolkit -# See: https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md -# Project source code URL: https://github.com/prometheus/prometheus - -matrix_prometheus_enabled: false - -matrix_prometheus_version: v2.42.0 -matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}" -matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}" - -matrix_prometheus_base_path: "{{ matrix_base_data_path }}/prometheus" -matrix_prometheus_config_path: "{{ matrix_prometheus_base_path }}/config" -matrix_prometheus_data_path: "{{ matrix_prometheus_base_path }}/data" - -# A list of extra arguments to pass to the container -matrix_prometheus_container_extra_arguments: [] - -# List of systemd services that matrix-prometheus.service depends on -matrix_prometheus_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-prometheus.service wants -matrix_prometheus_systemd_wanted_services_list: [] - -# The base container network. It will be auto-created by this role if it doesn't exist already. -matrix_prometheus_container_network: '' - -# A list of additional container networks that the container would be connected to. -# The role does not create these networks, so make sure they already exist. -# Use this to expose this container to another reverse proxy, which runs in a different container network. -matrix_prometheus_container_additional_networks: [] - -# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9090 in the container). -# -# Takes an ":" or "" value (e.g. "127.0.0.1:9090"), or empty string to not expose. -matrix_prometheus_container_http_host_bind_port: '' - -# A list of default arguments to pass to the prometheus process -matrix_prometheus_process_default_arguments: - - "--config.file=/etc/prometheus/prometheus.yml" - - "--storage.tsdb.path=/prometheus" - - "--web.console.libraries=/usr/share/prometheus/console_libraries" - - "--web.console.templates=/usr/share/prometheus/consoles" - -# A list of extra arguments to pass to the prometheus process -matrix_prometheus_process_extra_arguments: [] - -# holds the final list of process arguments -matrix_prometheus_process_arguments: "{{ matrix_prometheus_process_default_arguments + matrix_prometheus_process_extra_arguments }}" - -# Tells whether the "synapse" scraper configuration is enabled. -matrix_prometheus_scraper_synapse_enabled: false - -# Tells whether to download and load a Synapse rules file -matrix_prometheus_scraper_synapse_rules_enabled: "{{ matrix_prometheus_scraper_synapse_enabled }}" -matrix_prometheus_scraper_synapse_rules_synapse_tag: "master" -matrix_prometheus_scraper_synapse_rules_download_url: "https://raw.githubusercontent.com/matrix-org/synapse/{{ matrix_prometheus_scraper_synapse_rules_synapse_tag }}/contrib/prometheus/synapse-v2.rules" - -matrix_prometheus_scraper_synapse_targets: [] -matrix_prometheus_scraper_synapse_workers_enabled_list: [] - -# Tells whether the "node" scraper configuration is enabled. -# This configuration aims to scrape the current node (this server). -matrix_prometheus_scraper_node_enabled: false - -# Target addresses for the "node" scraper configuration. -# Unless you define this as a non-empty list, it gets populated at runtime with the IP address of `matrix-prometheus-node-exporter` and port 9100. -matrix_prometheus_scraper_node_targets: [] - -# Default prometheus configuration template which covers the generic use case. -# You can customize it by controlling the various variables inside it. -# -# For a more advanced customization, you can extend the default (see `matrix_prometheus_configuration_extension_yaml`) -# or completely replace this variable with your own template. -matrix_prometheus_configuration_yaml: "{{ lookup('template', 'templates/prometheus.yml.j2') }}" - -matrix_prometheus_configuration_extension_yaml: | - # Your custom YAML configuration goes here. - # This configuration extends the default starting configuration (`matrix_prometheus_configuration_yaml`). - # - # You can override individual variables from the default configuration, or introduce new ones. - # - # If you need something more special, you can take full control by - # completely redefining `matrix_prometheus_configuration_yaml`. - -matrix_prometheus_configuration_extension: "{{ matrix_prometheus_configuration_extension_yaml | from_yaml if matrix_prometheus_configuration_extension_yaml | from_yaml is mapping else {} }}" - -# Holds the final configuration (a combination of the default and its extension). -# You most likely don't need to touch this variable. Instead, see `matrix_prometheus_configuration_yaml`. -matrix_prometheus_configuration: "{{ matrix_prometheus_configuration_yaml | from_yaml | combine(matrix_prometheus_configuration_extension, recursive=True) }}" diff --git a/roles/custom/matrix-prometheus/tasks/main.yml b/roles/custom/matrix-prometheus/tasks/main.yml deleted file mode 100644 index bbd82001..00000000 --- a/roles/custom/matrix-prometheus/tasks/main.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- tags: - - setup-all - - setup-prometheus - - install-all - - install-prometheus - block: - - when: matrix_prometheus_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - - - when: matrix_prometheus_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - -- tags: - - setup-all - - setup-prometheus - block: - - when: not matrix_prometheus_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" diff --git a/roles/custom/matrix-prometheus/tasks/setup_install.yml b/roles/custom/matrix-prometheus/tasks/setup_install.yml deleted file mode 100644 index f606e268..00000000 --- a/roles/custom/matrix-prometheus/tasks/setup_install.yml +++ /dev/null @@ -1,57 +0,0 @@ ---- - -- name: Ensure matrix-prometheus image is pulled - community.docker.docker_image: - name: "{{ matrix_prometheus_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_prometheus_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure Prometheus paths exists - ansible.builtin.file: - path: "{{ item }}" - state: directory - mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - with_items: - - "{{ matrix_prometheus_base_path }}" - - "{{ matrix_prometheus_config_path }}" - - "{{ matrix_prometheus_data_path }}" - -- name: Download synapse-v2.rules - ansible.builtin.get_url: - url: "{{ matrix_prometheus_scraper_synapse_rules_download_url }}" - dest: "{{ matrix_prometheus_config_path }}/synapse-v2.rules" - force: true - mode: 0440 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - when: "matrix_prometheus_scraper_synapse_rules_enabled | bool" - register: result - retries: "{{ devture_playbook_help_geturl_retries_count }}" - delay: "{{ devture_playbook_help_geturl_retries_delay }}" - until: result is not failed - -- name: Ensure prometheus.yml installed - ansible.builtin.copy: - content: "{{ matrix_prometheus_configuration | to_nice_yaml(indent=2, width=999999) }}" - dest: "{{ matrix_prometheus_config_path }}/prometheus.yml" - mode: 0644 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - -- name: Ensure Prometheus container network is created - community.general.docker_network: - name: "{{ matrix_prometheus_container_network }}" - driver: bridge - -- name: Ensure matrix-prometheus.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-prometheus.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus.service" - mode: 0644 diff --git a/roles/custom/matrix-prometheus/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus/tasks/setup_uninstall.yml deleted file mode 100644 index 9d13d216..00000000 --- a/roles/custom/matrix-prometheus/tasks/setup_uninstall.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- name: Check existence of matrix-prometheus service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus.service" - register: matrix_prometheus_service_stat - -- when: matrix_prometheus_service_stat.stat.exists | bool - block: - - name: Ensure matrix-prometheus is stopped - ansible.builtin.service: - name: matrix-prometheus - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-prometheus.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus.service" - state: absent diff --git a/roles/custom/matrix-prometheus/tasks/validate_config.yml b/roles/custom/matrix-prometheus/tasks/validate_config.yml deleted file mode 100644 index 49121e7e..00000000 --- a/roles/custom/matrix-prometheus/tasks/validate_config.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- - -- name: Fail if Synapse metrics or Prometheus Node Exporter not enabled - ansible.builtin.fail: - msg: > - You need to enable `matrix_prometheus_scraper_synapse_enabled` and/or `matrix_prometheus_scraper_node_enabled` for Prometheus grab metrics. - when: "not matrix_prometheus_scraper_synapse_enabled and not matrix_prometheus_scraper_node_enabled" - -- name: Fail if required Prometheus settings not defined - ansible.builtin.fail: - msg: > - You need to define a required configuration setting (`{{ item }}`). - when: "vars[item] == ''" - with_items: - - matrix_prometheus_container_network diff --git a/roles/custom/matrix-prometheus/templates/prometheus.yml.j2 b/roles/custom/matrix-prometheus/templates/prometheus.yml.j2 deleted file mode 100644 index 6ed7cd92..00000000 --- a/roles/custom/matrix-prometheus/templates/prometheus.yml.j2 +++ /dev/null @@ -1,73 +0,0 @@ -#jinja2: lstrip_blocks: "True" -global: - scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. - evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. - # scrape_timeout is set to the global default (10s). - -# Load rules once and periodically evaluate them according to the global 'evaluation_interval'. -rule_files: - {% if matrix_prometheus_scraper_synapse_rules_enabled %} - - 'synapse-v2.rules' - {% endif %} - -# A scrape configuration containing exactly one endpoint to scrape: -# Here it's Prometheus itself. -scrape_configs: - # The job name is added as a label `job=` to any timeseries scraped from this config. - - job_name: 'prometheus' - - # Override the global default and scrape targets from this job every 5 seconds. - scrape_interval: 5s - scrape_timeout: 5s - - # metrics_path defaults to '/metrics' - # scheme defaults to 'http'. - - static_configs: - - targets: ['localhost:9090'] - - {% if matrix_prometheus_scraper_synapse_enabled %} - - job_name: 'synapse' - metrics_path: '/_synapse/metrics' - static_configs: - - targets: {{ matrix_prometheus_scraper_synapse_targets|to_json }} - labels: - instance: {{ matrix_domain | to_json }} - job: master - index: 0 - {% for worker in matrix_prometheus_scraper_synapse_workers_enabled_list %} - {% if worker.metrics_port != 0 %} - - targets: ['{{ worker.name }}:{{ worker.metrics_port }}'] - labels: - instance: {{ matrix_domain | to_json }} - worker_id: {{ worker.id | to_json }} - job: {{ worker.type | to_json }} - app: {{ worker.app | to_json }} - {% endif %} - {% endfor %} - {% endif %} - - {% if matrix_prometheus_scraper_node_enabled %} - - job_name: node - static_configs: - - targets: {{ matrix_prometheus_scraper_node_targets|to_json }} - {% endif %} - - {% if matrix_prometheus_scraper_postgres_enabled %} - - job_name: postgres - static_configs: - - targets: {{ matrix_prometheus_scraper_postgres_targets|to_json }} - {% endif %} - - {% if matrix_prometheus_scraper_hookshot_enabled %} - - job_name: hookshot - static_configs: - - targets: {{ matrix_prometheus_scraper_hookshot_targets|to_json }} - {% endif %} - - {% if matrix_prometheus_scraper_nginxlog_enabled %} - - job_name: nginxlog - static_configs: - - targets: - - {{ matrix_prometheus_scraper_nginxlog_server_port}} - {% endif %} diff --git a/roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 b/roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 deleted file mode 100644 index 2b716db6..00000000 --- a/roles/custom/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 +++ /dev/null @@ -1,51 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=matrix-prometheus -{% for service in matrix_prometheus_systemd_required_services_list %} -Requires={{ service }} -After={{ service }} -{% endfor %} -{% for service in matrix_prometheus_systemd_wanted_services_list %} -Wants={{ service }} -{% endfor %} -DefaultDependencies=no - -[Service] -Type=simple -Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus 2>/dev/null || true' - -ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ - --rm \ - --name=matrix-prometheus \ - --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --cap-drop=ALL \ - --read-only \ - --network={{ matrix_prometheus_container_network }} \ - {% if matrix_prometheus_container_http_host_bind_port %} - -p {{ matrix_prometheus_container_http_host_bind_port }}:9090 \ - {% endif %} - -v {{ matrix_prometheus_config_path }}:/etc/prometheus:z \ - -v {{ matrix_prometheus_data_path }}:/prometheus:z \ - {% for arg in matrix_prometheus_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - {{ matrix_prometheus_docker_image }} {{ matrix_prometheus_process_arguments|join(' ') }} - -{% for network in matrix_prometheus_container_additional_networks %} -ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-prometheus -{% endfor %} - -ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-prometheus - -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus 2>/dev/null || true' - -Restart=always -RestartSec=30 -SyslogIdentifier=matrix-prometheus - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index 9c958505..250b3e96 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -39,6 +39,25 @@ - {'old': 'matrix_playbook_traefik_certs_dumper_role_enabled', 'new': 'devture_traefik_certs_dumper_enabled'} - {'old': 'matrix_playbook_traefik_role_enabled', 'new': 'devture_traefik_enabled'} + # Most important variables from the old Prometheus role here, for people who miss the changelog entry + - {'old': 'matrix_prometheus_enabled', 'new': 'prometheus_enabled'} + - {'old': 'matrix_prometheus_process_default_arguments', 'new': 'prometheus_process_default_arguments'} + - {'old': 'matrix_prometheus_process_extra_arguments', 'new': 'prometheus_process_extra_arguments'} + - {'old': 'matrix_prometheus_configuration_extension_yaml', 'new': 'prometheus_configuration_extension_yaml'} + - {'old': 'matrix_prometheus_scraper_synapse_enabled', 'new': 'matrix_prometheus_services_connect_synapse_enabled'} + - {'old': 'matrix_prometheus_scraper_synapse_targets', 'new': ''} + - {'old': 'matrix_prometheus_scraper_synapse_workers_enabled_list', 'new': 'matrix_prometheus_services_connect_scraper_synapse_workers_enabled_list'} + - {'old': 'matrix_prometheus_scraper_synapse_rules_synapse_tag', 'new': 'matrix_prometheus_services_connect_synapse_rules_download_synapse_tag'} + - {'old': 'matrix_prometheus_scraper_node_enabled', 'new': 'prometheus_self_node_scraper_enabled'} + - {'old': 'matrix_prometheus_scraper_node_targets', 'new': ''} + - {'old': 'matrix_prometheus_scraper_postgres_enabled', 'new': 'matrix_prometheus_services_connect_scraper_postgres_enabled'} + - {'old': 'matrix_prometheus_scraper_postgres_targets', 'new': ''} + - {'old': 'matrix_prometheus_scraper_hookshot_enabled', 'new': 'matrix_prometheus_services_connect_scraper_hookshot_enabled'} + - {'old': 'matrix_prometheus_scraper_hookshot_targets', 'new': ''} + - {'old': 'matrix_prometheus_scraper_nginxlog_enabled', 'new': 'matrix_prometheus_services_connect_scraper_nginxlog_enabled'} + - {'old': 'matrix_prometheus_scraper_nginxlog_server_port', 'new': 'matrix_prometheus_services_connect_scraper_nginxlog_static_configs_target'} + - {'old': 'XXXXX', 'new': 'XXXXX'} + - name: (Deprecation) Catch and report matrix_postgres variables ansible.builtin.fail: msg: |- From 5c0cc0168cb0222f8f67624c457b02f20886d2db Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 21 Mar 2023 17:59:58 +0000 Subject: [PATCH 364/382] Update prometheus v2.42.0 -> v2.43.0 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index ebd3fd0f..23cb3a98 100644 --- a/requirements.yml +++ b/requirements.yml @@ -66,4 +66,4 @@ - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus.git name: prometheus - version: v2.42.0-0 + version: v2.43.0-0 From 38fc864745f438f935e8287315a478d673ac1813 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 21 Mar 2023 18:00:38 +0000 Subject: [PATCH 365/382] Update hookshot 3.0.0 -> 3.0.1 --- roles/custom/matrix-bridge-hookshot/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-bridge-hookshot/defaults/main.yml b/roles/custom/matrix-bridge-hookshot/defaults/main.yml index 2113f53a..17567c3b 100644 --- a/roles/custom/matrix-bridge-hookshot/defaults/main.yml +++ b/roles/custom/matrix-bridge-hookshot/defaults/main.yml @@ -10,7 +10,7 @@ matrix_hookshot_container_image_self_build: false matrix_hookshot_container_image_self_build_repo: "https://github.com/matrix-org/matrix-hookshot.git" matrix_hookshot_container_image_self_build_branch: "{{ 'main' if matrix_hookshot_version == 'latest' else matrix_hookshot_version }}" -matrix_hookshot_version: 3.0.0 +matrix_hookshot_version: 3.0.1 matrix_hookshot_docker_image: "{{ matrix_hookshot_docker_image_name_prefix }}halfshot/matrix-hookshot:{{ matrix_hookshot_version }}" matrix_hookshot_docker_image_name_prefix: "{{ 'localhost/' if matrix_hookshot_container_image_self_build else matrix_container_global_registry_prefix }}" From 045542be764997d43bbce6f6116c15ffe9427d13 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Tue, 21 Mar 2023 18:03:53 +0000 Subject: [PATCH 366/382] Update redis 7.0.9 -> 7.0.10 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index ebd3fd0f..d6578fb2 100644 --- a/requirements.yml +++ b/requirements.yml @@ -46,7 +46,7 @@ version: v2.2.0-0 - src: git+https://gitlab.com/etke.cc/roles/redis.git - version: v7.0.9-0 + version: v7.0.10-0 - src: git+https://gitlab.com/etke.cc/roles/etherpad.git version: v1.8.18-2 From 7a7c75c6a442e73a73aab4e7784e5218f336a2f9 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Mar 2023 07:43:47 +0200 Subject: [PATCH 367/382] Upgrade prometheus_postgres_exporter (v0.11.1-2 -> v0.12.0-0) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 69c9f8e7..d395410d 100644 --- a/requirements.yml +++ b/requirements.yml @@ -34,7 +34,7 @@ version: v1.5.0-6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git - version: v0.11.1-2 + version: v0.12.0-0 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git version: v1.2.3-1.7.9-0 From c222391e7756e064c319d3172d9876c18840f170 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Mar 2023 07:46:53 +0200 Subject: [PATCH 368/382] Upgrade Traefik (v2.9.8-2 -> v2.9.9-0) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index d395410d..91d16dae 100644 --- a/requirements.yml +++ b/requirements.yml @@ -55,7 +55,7 @@ version: v0.1.1-1 - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git - version: v2.9.8-2 + version: v2.9.9-0 - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: v2.8.1-0 From 69b2df629b5020a13c1c14ee7d98d0f9beefd1d8 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Mar 2023 08:04:25 +0200 Subject: [PATCH 369/382] Enable some recommended Coturn options in an effort to lower DDoS amplification factor Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2592 --- roles/custom/matrix-coturn/defaults/main.yml | 14 ++++++++++++++ .../matrix-coturn/templates/turnserver.conf.j2 | 12 ++++++++++++ 2 files changed, 26 insertions(+) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 9922f1d0..ee662b25 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -112,11 +112,25 @@ matrix_coturn_user_quota: null matrix_coturn_total_quota: null # Controls whether `no-tcp-relay` is added to the configuration +# Learn more here: https://github.com/coturn/coturn/blob/242eb78227f66442ba1573c00ec4552faae23eed/examples/etc/turnserver.conf#L419-L422 matrix_coturn_no_tcp_relay_enabled: true # Controls whether `no-multicast-peers` is added to the configuration +# Learn more here: https://github.com/coturn/coturn/blob/242eb78227f66442ba1573c00ec4552faae23eed/examples/etc/turnserver.conf#L629-L632 matrix_coturn_no_multicast_peers_enabled: true +# Controls whether `no-rfc5780` is added to the configuration +# Learn more here: https://github.com/coturn/coturn/blob/242eb78227f66442ba1573c00ec4552faae23eed/examples/etc/turnserver.conf#L770-L781 +matrix_coturn_no_rfc5780_enabled: true + +# Controls whether `no-stun-backward-compatibility` is added to the configuration +# Learn more here: https://github.com/coturn/coturn/blob/242eb78227f66442ba1573c00ec4552faae23eed/examples/etc/turnserver.conf#L783-L789 +matrix_coturn_no_stun_backward_compatibility_enabled: true + +# Controls whether `response-origin-only-with-rfc5780` is added to the configuration +# Learn more here: https://github.com/coturn/coturn/blob/242eb78227f66442ba1573c00ec4552faae23eed/examples/etc/turnserver.conf#L791-L796 +matrix_coturn_response_origin_only_with_rfc5780_enabled: true + # Additional configuration to be passed to turnserver.conf # Example: # matrix_coturn_additional_configuration: | diff --git a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 index dfc3a87d..3ed7b99f 100644 --- a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 +++ b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 @@ -35,6 +35,18 @@ prod no-tcp-relay {% endif %} +{% if matrix_coturn_no_rfc5780_enabled %} +no-rfc5780 +{% endif %} + +{% if matrix_coturn_no_stun_backward_compatibility_enabled %} +no-stun-backward-compatibility +{% endif %} + +{% if matrix_coturn_response_origin_only_with_rfc5780_enabled %} +response-origin-only-with-rfc5780 +{% endif %} + {% if matrix_coturn_user_quota != None %} user-quota={{ matrix_coturn_user_quota }} {% endif %} From 87b74e6c3e01f488b2ade1e1998f4e737c73b842 Mon Sep 17 00:00:00 2001 From: Felix Stupp Date: Wed, 22 Mar 2023 09:14:36 +0000 Subject: [PATCH 370/382] ntfy doc: Describe web app & how to enable it See #2529 --- docs/configuring-playbook-ntfy.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/configuring-playbook-ntfy.md b/docs/configuring-playbook-ntfy.md index 27d2fb12..1311afc9 100644 --- a/docs/configuring-playbook-ntfy.md +++ b/docs/configuring-playbook-ntfy.md @@ -21,6 +21,9 @@ ntfy_enabled: true # Uncomment the line below and change it, if you'd like. # matrix_server_fqn_ntfy: "ntfy.{{ matrix_domain }}" +# Uncomment to enable the ntfy web app (disabled by default) +# ntfy_web_root: app # defaults to "disable" + # Uncomment and change to inject additional configuration options. # ntfy_configuration_extension_yaml: | # log_level: DEBUG @@ -81,6 +84,12 @@ If the matrix app asks, "Choose a distributor: FCM Fallback or ntfy", then choos If the matrix app doesn't seem to pick it up, try restarting it and try the Troubleshooting section below. +### Web App + +ntfy has also an web app to subscribe and push to topics from the browser. This may be helpful to further troubleshoot UnifiedPush problems or to use ntfy for other purposes. The web app only runs in the browser locally (after downloading the JavaScript). + +The web app is disabled in this playbook by default as the expectation is that most users won't use it. You can either use the [official hosted one](https://ntfy.sh/app) (it supports using other public reachable ntfy instances) or host it yourself by setting `ntfy_web_root: "app"` and re-running Ansible. + ## Troubleshooting From e29af490c08b2b651230158738423b3c1dec95f2 Mon Sep 17 00:00:00 2001 From: Felix Stupp Date: Wed, 22 Mar 2023 09:21:29 +0000 Subject: [PATCH 371/382] CHANGELOG: Add disabled ntfy web app See #2529 --- CHANGELOG.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c9de0ac..7a7995e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,12 @@ +# 2023-03-22 + +## ntfy Web App is disabled by default + +ntfy provides a web app, which is now disabled by default, because it may be unknown to and unused by most users of this playbook. You can enable it by setting `ntfy_web_root: "app"` (see [ntfy documentation](docs/configuring-playbook-ntfy.md)). + +This change was already applied a while before this entry, but as some users were reporting the missing web app, this entry was added (see [#2529](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2529)). + + # 2023-03-21 ## The matrix-prometheus role lives independently now From a54d8f82a0b7923f5ac9234f92c8eb34a8a97541 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Mar 2023 11:28:34 +0200 Subject: [PATCH 372/382] Update configuring-playbook-ntfy.md --- docs/configuring-playbook-ntfy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-playbook-ntfy.md b/docs/configuring-playbook-ntfy.md index 1311afc9..c5549dbf 100644 --- a/docs/configuring-playbook-ntfy.md +++ b/docs/configuring-playbook-ntfy.md @@ -86,7 +86,7 @@ If the matrix app doesn't seem to pick it up, try restarting it and try the Trou ### Web App -ntfy has also an web app to subscribe and push to topics from the browser. This may be helpful to further troubleshoot UnifiedPush problems or to use ntfy for other purposes. The web app only runs in the browser locally (after downloading the JavaScript). +ntfy also has a web app to subscribe to and push to topics from the browser. This may be helpful to further troubleshoot UnifiedPush problems or to use ntfy for other purposes. The web app only runs in the browser locally (after downloading the JavaScript). The web app is disabled in this playbook by default as the expectation is that most users won't use it. You can either use the [official hosted one](https://ntfy.sh/app) (it supports using other public reachable ntfy instances) or host it yourself by setting `ntfy_web_root: "app"` and re-running Ansible. From 7be8943b642960b2ef3759a3df8d441c4838909a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 22 Mar 2023 17:53:57 +0200 Subject: [PATCH 373/382] Pin matrix-rageshake to 1.9.0 Related to: - https://github.com/matrix-org/rageshake/issues/69 - https://github.com/matrix-org/rageshake/pull/70 --- roles/custom/matrix-rageshake/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/custom/matrix-rageshake/defaults/main.yml b/roles/custom/matrix-rageshake/defaults/main.yml index 2dca91b1..8cc2f905 100644 --- a/roles/custom/matrix-rageshake/defaults/main.yml +++ b/roles/custom/matrix-rageshake/defaults/main.yml @@ -16,7 +16,7 @@ matrix_rageshake_path_prefix: / # There are no stable container image tags yet. # See: https://github.com/matrix-org/rageshake/issues/69 -matrix_rageshake_version: "master" +matrix_rageshake_version: 1.9.0 matrix_rageshake_base_path: "{{ matrix_base_data_path }}/rageshake" matrix_rageshake_config_path: "{{ matrix_rageshake_base_path }}/config" From 1a23016570185b9f01ca209a8559fbb72330a460 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Wed, 22 Mar 2023 16:54:39 +0000 Subject: [PATCH 374/382] Update grafana v9.4.3 -> v9.4.7 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 91d16dae..c3d45f85 100644 --- a/requirements.yml +++ b/requirements.yml @@ -40,7 +40,7 @@ version: v1.2.3-1.7.9-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git - version: v9.4.3-0 + version: v9.4.7-0 - src: git+https://gitlab.com/etke.cc/roles/ntfy.git version: v2.2.0-0 From 6b3ab052f6bb76ee8cf7f5339db6357785e28246 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 23 Mar 2023 11:02:20 +0200 Subject: [PATCH 375/382] Upgrade prometheus_node_exporter (v1.5.0-6 -> v1.5.0-7) --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index c3d45f85..248b98b1 100644 --- a/requirements.yml +++ b/requirements.yml @@ -31,7 +31,7 @@ version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git - version: v1.5.0-6 + version: v1.5.0-7 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.12.0-0 From 0b18f03195367f5f9de9f8e8235fc683bc8006a1 Mon Sep 17 00:00:00 2001 From: Aine Date: Thu, 23 Mar 2023 19:06:16 +0200 Subject: [PATCH 376/382] honoroit - add proper networking configuration and traefik labels --- group_vars/matrix_servers | 21 +++++++++ .../matrix-bot-honoroit/defaults/main.yml | 45 +++++++++++++++++++ .../tasks/setup_install.yml | 9 ++-- .../matrix-bot-honoroit/templates/labels.j2 | 45 +++++++++++++++++++ .../systemd/matrix-bot-honoroit.service.j2 | 13 +++++- 5 files changed, 128 insertions(+), 5 deletions(-) create mode 100644 roles/custom/matrix-bot-honoroit/templates/labels.j2 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 0b8d7e25..bb892a8b 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1671,6 +1671,27 @@ matrix_bot_honoroit_database_hostname: "{{ devture_postgres_connection_hostname matrix_bot_honoroit_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'honoroit.bot.db', rounds=655555) | to_uuid }}" matrix_bot_honoroit_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm64'] }}" +matrix_bot_honoroit_container_network: "{{ matrix_docker_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-bot-honoroit' }}" + +# For now, we attach this to the matrix-nginx-proxy network (unless that's the same network as the main one for us), +# because that's where the homeserver is expected to be accessed from. +# In the future, this will need to be redone. +matrix_bot_honoroit_container_additional_networks: | + {{ + ( + ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else []) + + + ([matrix_nginx_proxy_container_network] if matrix_nginx_proxy_enabled and matrix_bot_honoroit_container_network != matrix_nginx_proxy_container_network else []) + + + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != matrix_bot_honoroit_container_network else []) + ) | unique + }} + +matrix_bot_honoroit_container_labels_traefik_enabled: "{{ matrix_playbook_traefik_labels_enabled }}" +matrix_bot_honoroit_container_labels_traefik_docker_network: "{{ matrix_playbook_reverse_proxyable_services_additional_network }}" +matrix_bot_honoroit_container_labels_traefik_entrypoints: "{{ devture_traefik_entrypoint_primary }}" +matrix_bot_honoroit_container_labels_traefik_tls_certResolver: "{{ devture_traefik_certResolver_primary }}" + ###################################################################### # # /matrix-bot-honoroit diff --git a/roles/custom/matrix-bot-honoroit/defaults/main.yml b/roles/custom/matrix-bot-honoroit/defaults/main.yml index 42f76682..38d02d34 100644 --- a/roles/custom/matrix-bot-honoroit/defaults/main.yml +++ b/roles/custom/matrix-bot-honoroit/defaults/main.yml @@ -4,6 +4,13 @@ matrix_bot_honoroit_enabled: true +# The hostname at which honoroit is served. +matrix_bot_honoroit_hostname: '' + +# The path at which honoroit is exposed. +# This value must either be `/` or not end with a slash (e.g. `/honoroit`). +matrix_bot_honoroit_path_prefix: / + matrix_bot_honoroit_container_image_self_build: false matrix_bot_honoroit_docker_repo: "https://gitlab.com/etke.cc/honoroit.git" matrix_bot_honoroit_docker_repo_version: "{{ matrix_bot_honoroit_version }}" @@ -19,6 +26,44 @@ matrix_bot_honoroit_config_path: "{{ matrix_bot_honoroit_base_path }}/config" matrix_bot_honoroit_data_path: "{{ matrix_bot_honoroit_base_path }}/data" matrix_bot_honoroit_data_store_path: "{{ matrix_bot_honoroit_data_path }}/store" +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_bot_honoroit_container_network: matrix-bot-honoroit + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_bot_honoroit_container_additional_networks: [] + +# matrix_bot_honoroit_container_labels_traefik_enabled controls whether labels to assist a Traefik reverse-proxy will be attached to the container. +# See `../templates/labels.j2` for details. +# +# To inject your own other container labels, see `matrix_bot_honoroit_container_labels_additional_labels`. +matrix_bot_honoroit_container_labels_traefik_enabled: true +matrix_bot_honoroit_container_labels_traefik_docker_network: "{{ matrix_bot_honoroit_container_network }}" +matrix_bot_honoroit_container_labels_traefik_hostname: "{{ matrix_bot_honoroit_hostname }}" +# The path prefix must either be `/` or not end with a slash (e.g. `/honoroit`). +matrix_bot_honoroit_container_labels_traefik_path_prefix: "{{ matrix_bot_honoroit_path_prefix }}" +matrix_bot_honoroit_container_labels_traefik_rule: "Host(`{{ matrix_bot_honoroit_container_labels_traefik_hostname }}`){% if matrix_bot_honoroit_container_labels_traefik_path_prefix != '/' %} && PathPrefix(`{{ matrix_bot_honoroit_container_labels_traefik_path_prefix }}`){% endif %}" +matrix_bot_honoroit_container_labels_traefik_priority: 0 +matrix_bot_honoroit_container_labels_traefik_entrypoints: web-secure +matrix_bot_honoroit_container_labels_traefik_tls: "{{ matrix_bot_honoroit_container_labels_traefik_entrypoints != 'web' }}" +matrix_bot_honoroit_container_labels_traefik_tls_certResolver: default # noqa var-naming + +# Controls which additional headers to attach to all HTTP responses. +# To add your own headers, use `matrix_bot_honoroit_container_labels_traefik_additional_response_headers_custom` +matrix_bot_honoroit_container_labels_traefik_additional_response_headers: "{{ matrix_bot_honoroit_container_labels_traefik_additional_response_headers_auto | combine(matrix_bot_honoroit_container_labels_traefik_additional_response_headers_custom) }}" +matrix_bot_honoroit_container_labels_traefik_additional_response_headers_auto: {} +matrix_bot_honoroit_container_labels_traefik_additional_response_headers_custom: {} + +# matrix_bot_buscarron_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# See `../templates/labels.j2` for details. +# +# Example: +# matrix_bot_buscarron_container_labels_additional_labels: | +# my.label=1 +# another.label="here" +matrix_bot_buscarron_container_labels_additional_labels: '' + # A list of extra arguments to pass to the container matrix_bot_honoroit_container_extra_arguments: [] diff --git a/roles/custom/matrix-bot-honoroit/tasks/setup_install.yml b/roles/custom/matrix-bot-honoroit/tasks/setup_install.yml index f7b264a8..14b5a0e4 100644 --- a/roles/custom/matrix-bot-honoroit/tasks/setup_install.yml +++ b/roles/custom/matrix-bot-honoroit/tasks/setup_install.yml @@ -40,13 +40,16 @@ - {path: "{{ matrix_bot_honoroit_docker_src_files_path }}", when: true} when: "item.when | bool" -- name: Ensure honoroit environment variables file created +- name: Ensure honoroit support files installed ansible.builtin.template: - src: "{{ role_path }}/templates/env.j2" - dest: "{{ matrix_bot_honoroit_config_path }}/env" + src: "{{ role_path }}/templates/{{ item }}.j2" + dest: "{{ matrix_bot_honoroit_config_path }}/{{ item }}" owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" mode: 0640 + with_items: + - env + - labels - name: Ensure honoroit image is pulled community.docker.docker_image: diff --git a/roles/custom/matrix-bot-honoroit/templates/labels.j2 b/roles/custom/matrix-bot-honoroit/templates/labels.j2 new file mode 100644 index 00000000..f03af34a --- /dev/null +++ b/roles/custom/matrix-bot-honoroit/templates/labels.j2 @@ -0,0 +1,45 @@ +{% if matrix_bot_honoroit_container_labels_traefik_enabled %} +traefik.enable=true + +{% if matrix_bot_honoroit_container_labels_traefik_docker_network %} +traefik.docker.network={{ matrix_bot_honoroit_container_labels_traefik_docker_network }} +{% endif %} + +{% set middlewares = [] %} + +{% if matrix_bot_honoroit_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-bot-honoroit-slashless-redirect.redirectregex.regex=({{ matrix_bot_honoroit_container_labels_traefik_path_prefix | quote }})$ +traefik.http.middlewares.matrix-bot-honoroit-slashless-redirect.redirectregex.replacement=${1}/ +{% set middlewares = middlewares + ['matrix-bot-honoroit-slashless-redirect'] %} +{% endif %} + +{% if matrix_bot_honoroit_container_labels_traefik_path_prefix != '/' %} +traefik.http.middlewares.matrix-bot-honoroit-strip-prefix.stripprefix.prefixes={{ matrix_bot_honoroit_container_labels_traefik_path_prefix }} +{% set middlewares = middlewares + ['matrix-bot-honoroit-strip-prefix'] %} +{% endif %} + +{% if matrix_bot_honoroit_container_labels_traefik_additional_response_headers.keys() | length > 0 %} +{% for name, value in matrix_bot_honoroit_container_labels_traefik_additional_response_headers.items() %} +traefik.http.middlewares.matrix-bot-honoroit-add-headers.headers.customresponseheaders.{{ name }}={{ value }} +{% endfor %} +{% set middlewares = middlewares + ['matrix-bot-honoroit-add-headers'] %} +{% endif %} + +traefik.http.routers.matrix-bot-honoroit.rule={{ matrix_bot_honoroit_container_labels_traefik_rule }} +{% if matrix_bot_honoroit_container_labels_traefik_priority | int > 0 %} +traefik.http.routers.matrix-bot-honoroit.priority={{ matrix_bot_honoroit_container_labels_traefik_priority }} +{% endif %} +traefik.http.routers.matrix-bot-honoroit.service=matrix-bot-honoroit +{% if middlewares | length > 0 %} +traefik.http.routers.matrix-bot-honoroit.middlewares={{ middlewares | join(',') }} +{% endif %} +traefik.http.routers.matrix-bot-honoroit.entrypoints={{ matrix_bot_honoroit_container_labels_traefik_entrypoints }} +traefik.http.routers.matrix-bot-honoroit.tls={{ matrix_bot_honoroit_container_labels_traefik_tls | to_json }} +{% if matrix_bot_honoroit_container_labels_traefik_tls %} +traefik.http.routers.matrix-bot-honoroit.tls.certResolver={{ matrix_bot_honoroit_container_labels_traefik_tls_certResolver }} +{% endif %} + +traefik.http.services.matrix-bot-honoroit.loadbalancer.server.port=8080 +{% endif %} + +{{ matrix_bot_honoroit_container_labels_additional_labels }} diff --git a/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 b/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 index 2cce62da..44fe9a2f 100644 --- a/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 +++ b/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 @@ -16,19 +16,28 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-honoroit 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-honoroit 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-bot-honoroit \ +ExecStart={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-bot-honoroit \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_bot_honoroit_container_network }} \ --env-file={{ matrix_bot_honoroit_config_path }}/env \ + --label-file={{ matrix_bot_honoroit_config_path }}/labels \ --mount type=bind,src={{ matrix_bot_honoroit_data_path }},dst=/data \ {% for arg in matrix_bot_honoroit_container_extra_arguments %} {{ arg }} \ {% endfor %} {{ matrix_bot_honoroit_docker_image }} +{% for network in matrix_bot_honoroit_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-bot-honoroit +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-bot-honoroit + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-honoroit 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-honoroit 2>/dev/null || true' Restart=always From 15ce37723561958b431a5ed9bdafacc979e3d0ec Mon Sep 17 00:00:00 2001 From: Aine Date: Thu, 23 Mar 2023 19:09:34 +0200 Subject: [PATCH 377/382] honoroit - add matrix_bot_honoroit_hostname into group vars --- group_vars/matrix_servers | 1 + 1 file changed, 1 insertion(+) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index bb892a8b..499a714b 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1653,6 +1653,7 @@ matrix_bot_maubot_database_password: "{{ '%s' | format(matrix_homeserver_generic # We don't enable bots by default. matrix_bot_honoroit_enabled: false +matrix_bot_honoroit_hostname: "{{ matrix_server_fqn_matrix }}" matrix_bot_honoroit_systemd_required_services_list: | {{ From 6d5cf9cbcd99e298352025edfd1b1eef539dbfc6 Mon Sep 17 00:00:00 2001 From: Aine Date: Thu, 23 Mar 2023 20:17:11 +0200 Subject: [PATCH 378/382] fix honoroit networks --- roles/custom/matrix-bot-honoroit/defaults/main.yml | 6 +++--- .../templates/systemd/matrix-bot-honoroit.service.j2 | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/roles/custom/matrix-bot-honoroit/defaults/main.yml b/roles/custom/matrix-bot-honoroit/defaults/main.yml index 38d02d34..5e90ef51 100644 --- a/roles/custom/matrix-bot-honoroit/defaults/main.yml +++ b/roles/custom/matrix-bot-honoroit/defaults/main.yml @@ -55,14 +55,14 @@ matrix_bot_honoroit_container_labels_traefik_additional_response_headers: "{{ ma matrix_bot_honoroit_container_labels_traefik_additional_response_headers_auto: {} matrix_bot_honoroit_container_labels_traefik_additional_response_headers_custom: {} -# matrix_bot_buscarron_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. +# matrix_bot_honoroit_container_labels_additional_labels contains a multiline string with additional labels to add to the container label file. # See `../templates/labels.j2` for details. # # Example: -# matrix_bot_buscarron_container_labels_additional_labels: | +# matrix_bot_honoroit_container_labels_additional_labels: | # my.label=1 # another.label="here" -matrix_bot_buscarron_container_labels_additional_labels: '' +matrix_bot_honoroit_container_labels_additional_labels: '' # A list of extra arguments to pass to the container matrix_bot_honoroit_container_extra_arguments: [] diff --git a/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 b/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 index 44fe9a2f..9bbc7d10 100644 --- a/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 +++ b/roles/custom/matrix-bot-honoroit/templates/systemd/matrix-bot-honoroit.service.j2 @@ -16,7 +16,7 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-honoroit 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-honoroit 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} create \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --rm \ --name=matrix-bot-honoroit \ --log-driver=none \ From e04894de7cd2a2aff05f8404caff6258b62edc09 Mon Sep 17 00:00:00 2001 From: Aine <97398200+etkecc@users.noreply.github.com> Date: Fri, 24 Mar 2023 07:54:06 +0000 Subject: [PATCH 379/382] Update borg 1.2.3 -> 1.2.4 --- requirements.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 248b98b1..1a31c130 100644 --- a/requirements.yml +++ b/requirements.yml @@ -37,7 +37,7 @@ version: v0.12.0-0 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git - version: v1.2.3-1.7.9-0 + version: v1.2.4-1.7.9-0 - src: git+https://gitlab.com/etke.cc/roles/grafana.git version: v9.4.7-0 From e60febc9ca4e5a07a2dda195954df52992663139 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 25 Mar 2023 08:56:12 +0200 Subject: [PATCH 380/382] Use git for fetching the geerlingguy.docker role With this change, all dependency roles are downloaded using the same mechanism (git), which makes life simpler for tools like https://gitlab.com/etke.cc/int/agru --- requirements.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/requirements.yml b/requirements.yml index 1a31c130..386c8763 100644 --- a/requirements.yml +++ b/requirements.yml @@ -1,6 +1,7 @@ --- -- src: geerlingguy.docker +- src: git+https://github.com/geerlingguy/ansible-role-docker + name: geerlingguy.docker version: 6.1.0 - src: git+https://github.com/devture/com.devture.ansible.role.docker_sdk_for_python.git From 34a03cf73284f29db0bcef265f8741d374561d95 Mon Sep 17 00:00:00 2001 From: Aine Date: Sat, 25 Mar 2023 17:47:13 +0200 Subject: [PATCH 381/382] safely integrate agru, format justfile, format requirements.yml --- justfile | 30 ++++++++++++++++++++---------- requirements.yml | 29 +++-------------------------- 2 files changed, 23 insertions(+), 36 deletions(-) diff --git a/justfile b/justfile index f50b5217..bcacd78c 100644 --- a/justfile +++ b/justfile @@ -1,48 +1,58 @@ # Shows help default: - @just --list --justfile {{ justfile() }} + @just --list --justfile {{ justfile() }} # Pulls external Ansible roles roles: - rm -rf roles/galaxy - ansible-galaxy install -r requirements.yml -p roles/galaxy/ --force + #!/usr/bin/env sh + set -euo pipefail + if [ -x "$(command -v agru)" ]; then + agru + else + rm -rf roles/galaxy + ansible-galaxy install -r requirements.yml -p roles/galaxy/ --force + fi + +# Updates requirements.yml if there are any new tags available. Requires agru +update: + @agru -u # Runs ansible-lint against all roles in the playbook lint: - ansible-lint + ansible-lint # Runs the playbook with --tags=install-all,ensure-matrix-users-created,start and optional arguments install-all *extra_args: (run-tags "install-all,ensure-matrix-users-created,start" extra_args) # Runs installation tasks for a single service install-service service *extra_args: - just --justfile {{ justfile() }} run --tags=install-{{ service }},start-group --extra-vars=group={{ service }} {{ extra_args }} + just --justfile {{ justfile() }} run --tags=install-{{ service }},start-group --extra-vars=group={{ service }} {{ extra_args }} # Runs the playbook with --tags=setup-all,ensure-matrix-users-created,start and optional arguments setup-all *extra_args: (run-tags "setup-all,ensure-matrix-users-created,start" extra_args) # Runs the playbook with the given list of arguments run +extra_args: - time ansible-playbook -i inventory/hosts setup.yml {{ extra_args }} + time ansible-playbook -i inventory/hosts setup.yml {{ extra_args }} # Runs the playbook with the given list of comma-separated tags and optional arguments run-tags tags *extra_args: - just --justfile {{ justfile() }} run --tags={{ tags }} {{ extra_args }} + just --justfile {{ justfile() }} run --tags={{ tags }} {{ extra_args }} # Runs the playbook in user-registration mode register-user username password admin_yes_or_no *extra_args: - time ansible-playbook -i inventory/hosts setup.yml --tags=register-user --extra-vars="username={{ username }} password={{ password }} admin={{ admin_yes_or_no }}" {{ extra_args }} + time ansible-playbook -i inventory/hosts setup.yml --tags=register-user --extra-vars="username={{ username }} password={{ password }} admin={{ admin_yes_or_no }}" {{ extra_args }} # Starts all services start-all *extra_args: (run-tags "start-all" extra_args) # Starts a specific service group start-group group *extra_args: - @just --justfile {{ justfile() }} run-tags start-group --extra-vars="group={{ group }}" {{ extra_args }} + @just --justfile {{ justfile() }} run-tags start-group --extra-vars="group={{ group }}" {{ extra_args }} # Stops all services stop-all *extra_args: (run-tags "stop-all" extra_args) # Stops a specific service group stop-group group *extra_args: - @just --justfile {{ justfile() }} run-tags stop-group --extra-vars="group={{ group }}" {{ extra_args }} + @just --justfile {{ justfile() }} run-tags stop-group --extra-vars="group={{ group }}" {{ extra_args }} diff --git a/requirements.yml b/requirements.yml index 386c8763..778e197a 100644 --- a/requirements.yml +++ b/requirements.yml @@ -1,70 +1,47 @@ ---- - - src: git+https://github.com/geerlingguy/ansible-role-docker - name: geerlingguy.docker version: 6.1.0 - + name: geerlingguy.docker - src: git+https://github.com/devture/com.devture.ansible.role.docker_sdk_for_python.git version: 129c8590e106b83e6f4c259649a613c6279e937a - - src: git+https://github.com/devture/com.devture.ansible.role.playbook_help.git version: c1f40e82b4d6b072b6f0e885239322bdaaaf554f - - src: git+https://github.com/devture/com.devture.ansible.role.systemd_docker_base.git version: 327d2e17f5189ac2480d6012f58cf64a2b46efba - - src: git+https://github.com/devture/com.devture.ansible.role.timesync.git version: 3d5bb2976815958cdce3f368fa34fb51554f899b - - src: git+https://github.com/devture/com.devture.ansible.role.playbook_state_preserver.git version: ff2fd42e1c1a9e28e3312bbd725395f9c2fc7f16 - - src: git+https://github.com/devture/com.devture.ansible.role.postgres.git version: 38764398bf82b06a1736c3bfedc71dfd229e4b52 - - src: git+https://github.com/devture/com.devture.ansible.role.postgres_backup.git version: 8e9ec48a09284c84704d7a2dce17da35f181574d - - src: git+https://github.com/devture/com.devture.ansible.role.systemd_service_manager.git version: 6ccb88ac5fc27e1e70afcd48278ade4b564a9096 - - src: git+https://github.com/devture/com.devture.ansible.role.playbook_runtime_messages.git version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git version: v1.5.0-7 - - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.12.0-0 - - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git version: v1.2.4-1.7.9-0 - - src: git+https://gitlab.com/etke.cc/roles/grafana.git version: v9.4.7-0 - - src: git+https://gitlab.com/etke.cc/roles/ntfy.git version: v2.2.0-0 - - src: git+https://gitlab.com/etke.cc/roles/redis.git version: v7.0.10-0 - - src: git+https://gitlab.com/etke.cc/roles/etherpad.git version: v1.8.18-2 - - src: git+https://github.com/devture/com.devture.ansible.role.container_socket_proxy.git version: v0.1.1-1 - - src: git+https://github.com/devture/com.devture.ansible.role.traefik.git version: v2.9.9-0 - - src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git version: v2.8.1-0 - - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-aux.git - name: aux version: v1.0.0-0 - + name: aux - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus.git - name: prometheus version: v2.43.0-0 + name: prometheus From 413049feea13fe5bece06ae0c331514b62d706d2 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sun, 26 Mar 2023 14:12:29 +0300 Subject: [PATCH 382/382] Remove shell configuration for better compatibility Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2604 --- justfile | 1 - 1 file changed, 1 deletion(-) diff --git a/justfile b/justfile index bcacd78c..6eac034e 100644 --- a/justfile +++ b/justfile @@ -5,7 +5,6 @@ default: # Pulls external Ansible roles roles: #!/usr/bin/env sh - set -euo pipefail if [ -x "$(command -v agru)" ]; then agru else