SelfPrivacy API

API available as Swagger documentation, markdown version is below.

Version

1.2.0

Content negotiation

URI Scheme: http

Consumes: application/json

Produces: application/json

Access control

• bearerAuth

All endpoints

backups

bitwarden

email

gitea

nextcloud

ocserv

pleroma

services

ssh

system

tokens

users

Paths

Delete token (DeleteAuthTokens)

DELETE /auth/tokens

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

DeleteAuthTokensBody

Properties

Delete SSH key (DeleteServicesSSHKeysUsername)

DELETE /services/ssh/keys/{username}

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

DeleteServicesSSHKeysUsernameBody

Properties

Delete a user (DeleteUsersUsername)

DELETE /users/{username}

Security Requirements

• bearerAuth

Parameters

All responses

Get API version (GetAPIVersion)

GET /api/version

All responses

Responses

200 - API version

Status: OK

Schema

401 - Unauthorized

Status: Unauthorized

Get recovery token status (GetAuthRecoveryToken)

GET /auth/recovery_token

Security Requirements

• bearerAuth

All responses

Responses

200 - Recovery token status

Status: OK

Schema

400 - Bad request

Status: Bad Request

Get current device tokens (GetAuthTokens)

GET /auth/tokens

Security Requirements

• bearerAuth

All responses

Get DKIM key from file (GetServicesMailserverDkim)

GET /services/mailserver/dkim

Security Requirements

• bearerAuth

All responses

Get all restic backups (GetServicesResticBackupList)

GET /services/restic/backup/list

Security Requirements

• bearerAuth

All responses

Force reload snapshots (GetServicesResticBackupReload)

GET /services/restic/backup/reload

Security Requirements

• bearerAuth

All responses

Get backup status (GetServicesResticBackupStatus)

GET /services/restic/backup/status

Security Requirements

• bearerAuth

All responses

Get current SSH settings (GetServicesSSH)

GET /services/ssh

Security Requirements

• bearerAuth

All responses

List SSH keys (GetServicesSSHKeysUsername)

GET /services/ssh/keys/{username}

Security Requirements

• bearerAuth

Parameters

All responses

Get service status (GetServicesStatus)

GET /services/status

All responses

Responses

200 - Service status

Status: OK

Schema

401 - Unauthorized

Status: Unauthorized

Rebuild NixOS with nixos-rebuild switch (GetSystemConfigurationApply)

GET /system/configuration/apply

Security Requirements

• bearerAuth

All responses

Get current system autoupgrade settings (GetSystemConfigurationAutoUpgrade)

GET /system/configuration/autoUpgrade

Security Requirements

• bearerAuth

All responses

Pull Repository Changes (GetSystemConfigurationPull)

GET /system/configuration/pull

Security Requirements

• bearerAuth

All responses

Rollback NixOS with nixos-rebuild switch --rollback (GetSystemConfigurationRollback)

GET /system/configuration/rollback

Security Requirements

• bearerAuth

All responses

Get current system timezone (GetSystemConfigurationTimezone)

GET /system/configuration/timezone

Security Requirements

• bearerAuth

All responses

Upgrade NixOS with nixos-rebuild switch --upgrade (GetSystemConfigurationUpgrade)

GET /system/configuration/upgrade

Security Requirements

• bearerAuth

All responses

Get python version used by this API (GetSystemPythonVersion)

GET /system/pythonVersion

Security Requirements

• bearerAuth

All responses

Reboot the system (GetSystemReboot)

GET /system/reboot

Security Requirements

• bearerAuth

All responses

Get system version from uname -a (GetSystemVersion)

GET /system/version

Security Requirements

• bearerAuth

All responses

Get a list of users (GetUsers)

GET /users

Security Requirements

• bearerAuth

All responses

Get new device token (PostAuthNewDevice)

POST /auth/new_device

Security Requirements

• bearerAuth

All responses

Authorize device (PostAuthNewDeviceAuthorize)

POST /auth/new_device/authorize

Parameters

All responses

Inlined models

PostAuthNewDeviceAuthorizeBody

Properties

Generate recovery token (PostAuthRecoveryToken)

POST /auth/recovery_token

Security Requirements

• bearerAuth

Parameters

All responses

Responses

200 - Recovery token generated

Status: OK

Schema

400 - Bad request

Status: Bad Request

Inlined models

PostAuthRecoveryTokenBody

Properties

Use recovery token (PostAuthRecoveryTokenUse)

POST /auth/recovery_token/use

Parameters

All responses

Responses

200 - Recovery token used

Status: OK

Schema

Inlined models

PostAuthRecoveryTokenUseBody

Properties

Refresh token (PostAuthTokens)

POST /auth/tokens

Security Requirements

• bearerAuth

All responses

Disable Bitwarden (PostServicesBitwardenDisable)

POST /services/bitwarden/disable

Security Requirements

• bearerAuth

All responses

Enable Bitwarden (PostServicesBitwardenEnable)

POST /services/bitwarden/enable

Security Requirements

• bearerAuth

All responses

Disable Gitea (PostServicesGiteaDisable)

POST /services/gitea/disable

Security Requirements

• bearerAuth

All responses

Enable Gitea (PostServicesGiteaEnable)

POST /services/gitea/enable

Security Requirements

• bearerAuth

All responses

Disable Nextcloud (PostServicesNextcloudDisable)

POST /services/nextcloud/disable

Security Requirements

• bearerAuth

All responses

Enable Nextcloud (PostServicesNextcloudEnable)

POST /services/nextcloud/enable

Security Requirements

• bearerAuth

All responses

Disable OCserv (PostServicesOcservDisable)

POST /services/ocserv/disable

Security Requirements

• bearerAuth

All responses

Enable OCserv (PostServicesOcservEnable)

POST /services/ocserv/enable

Security Requirements

• bearerAuth

All responses

Disable Pleroma (PostServicesPleromaDisable)

POST /services/pleroma/disable

Security Requirements

• bearerAuth

All responses

Enable Pleroma (PostServicesPleromaEnable)

POST /services/pleroma/enable

Security Requirements

• bearerAuth

All responses

Enable SSH (PostServicesSSHEnable)

POST /services/ssh/enable

Security Requirements

• bearerAuth

All responses

Add SSH key to the user (PostServicesSSHKeysUsername)

POST /services/ssh/keys/{username}

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PostServicesSSHKeysUsernameBody

Properties

Create a new user (PostUsers)

POST /users

Consumes

• application/json

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PostUsersBody

Properties

Set the new key for backblaze (PutServicesResticBackblazeConfig)

PUT /services/restic/backblaze/config

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PutServicesResticBackblazeConfigBody

Properties

Initiate a new restic backup (PutServicesResticBackupCreate)

PUT /services/restic/backup/create

Security Requirements

• bearerAuth

All responses

Start backup restoration (PutServicesResticBackupRestore)

PUT /services/restic/backup/restore

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PutServicesResticBackupRestoreBody

Properties

Change SSH settings (PutServicesSSH)

PUT /services/ssh

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PutServicesSSHBody

Properties

Add a SSH root key (PutServicesSSHKeySend)

PUT /services/ssh/key/send

Consumes

• application/json

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PutServicesSSHKeySendBody

Properties

Change system auto upgrade settings (PutSystemConfigurationAutoUpgrade)

PUT /system/configuration/autoUpgrade

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PutSystemConfigurationAutoUpgradeBody

Properties

Change system timezone (PutSystemConfigurationTimezone)

PUT /system/configuration/timezone

Security Requirements

• bearerAuth

Parameters

All responses

Inlined models

PutSystemConfigurationTimezoneBody Properties

Changelog

1.2.0

• Added authorization tokens module to allow
  • Having more than one device controlling the server
  • Refreshing the current token
  • Obtaining a recovery token

API changes

• /auth/tokens added
• /auth/new_device added
• Public endpoint /auth/new_device/authorize added.
• /auth/recovery_token added
• Public endpoint /auth/recovery_token/use added.

New migrations

• [[migrations#Create tokens JSON file|Migrate to new tokens storage]]

1.1.1

Released on 26 Jan 2022

• Added [[migrations|migration module]] which handles one-shot actions which cannot be done by nixos-configuration.
• Added [[migrations#Fix NixOS configuration branch|migration]] which changes rolling-testing branch to master, bug introduced in 0.4.0 version of the app.

1.1.0

Released on 9 Dec 2021

• Added system configuration endpoints.
• SSH management.
• Backups controller.
• Systemd is now used for system rebuilds.
• WriteUserData and ReadUserData utils added to reduce boilerplate of file locking.
• Unit tests are now a thing.

API changes

• /system/upgrade is now /system/configuration/upgrade.
• /api/version added.
• /services/ssh added.
• /services/ssh/<username> added.
• /configuration/timezone added.
• /configuration/autoUpgrade added.
• /restic/backblaze/config added.
• /system/configuration/pull added.
• /services/restic/backup/reload added.

Pre-1.1.0 changes

This is an API refactoring stage. There was no endpoint to retrieve API version, but these all were referenced as 1.1.0 version in code.

Input sanitization, added swagger

Released on 17 Nov 2021

• Addressed [[vulnerabilities#SPCVE-0001|SPCVE-0001]].
• Added Swagger documentation.

Add basic API auth

Released on 16 Nov 2021

• Added basic token auth.

Move to JSON controlled server settings

Released on 16 Nov 2021

• Assumes server already moved to JSON-controlled Nix config.
• System is now configured by usage of userdata.json.
• File locking is used now.

Decomposition

Released on 12 Nov 2021

API changes

• /systemVersion is now /system/version.
• /getDKIM is now /services/mailserver/dkim.
• /pythonVersion is now /system/version/pythonVersion.
• /users/create is now POST on /users.
• /users/deleteUser is now DELETE on /users.
• /services/SERVICE_NAME/enable and /services/SERVICE_NAME/disable now also return message in their responses.
• /services/ssh/key/send: response field result renamed to status.

Other changes

• Moved to Python 3.9.
• Moved to flask_restful's Resource objects.
• distutils is deprecated, moved to setuptools.

Code formatting

Black is used for code formatting.

No linting rules applied yet.

Authorization

Before 1.2.0

At that time, only one access token could be used. It is declared during nixos-infect stage and generated by the mobile app.

After 1.2.0

New auth system was introduced in 1.2.0. See [[migrations#Create tokens JSON file]] for details on migration to the new system.

Tokens storage

Tokens are stored in /etc/nixos/userdata/tokens.json, which can be accessed only by root. File follows this schema:

{
    "$schema": "http://json-schema.org/schema#",
    "$id": "https://git.selfprivacy.org/inex/selfprivacy-nixos-config/raw/branch/master/userdata/tokens_schema.json",
    "type": "object",
    "properties": {
        "tokens": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "token": {
                        "type": "string"
                    },
                    "name": {
                        "type": "string"
                    },
                    "date": {
                        "type": "string"
                    }
                },
                "required": [
                    "token",
                    "name",
                    "date"
                ]
            }
        },
        "recovery_token": {
            "type": "object",
            "properties": {
                "token": {
                    "type": "string"
                },
                "date": {
                    "type": "string"
                },
                "expiration": {
                    "type": "string"
                },
                "uses_left": {
                    "type": "integer"
                }
            },
            "required": [
                "token",
                "date"
            ]
        },
        "new_device": {
            "type": "object",
            "properties": {
                "token": {
                    "type": "string"
                },
                "date": {
                    "type": "string"
                },
                "expiration": {
                    "type": "string"
                }
            },
            "required": [
                "token",
                "date",
                "expiration"
            ]
        }
    },
    "required": [
        "tokens"
    ]
}

i.e.:

{
    "tokens": [
        {
            "token": "device token",
            "name": "device name",
            "date": "date of creation",
        }
    ],
    "recovery_token": {
        "token": "recovery token",
        "date": "date of creation",
        "expiration": "date of expiration",
        "uses_left": "number of uses left"
    },
    "new_device": {
        "token": "new device auth token",
        "date": "date of creation",
        "expiration": "date of expiration",
    }
}

All dates MUST follow the %Y-%m-%dT%H:%M:%S.%fZ format.

Token control

Refer to the API documentation. You can list the tokens and delete tokens.

New device token creation

To authorize a new device, existing authorized device must acquire a new device token.

POST /auth/new_device is used without any arguments.

New device tokens are 16 random bytes encoded to the mnemonic phrase. Token lifetime is 10 minutes or less. There can only be one token for new device auth.

This token is used by the new device to get a proper access token.

POST /auth/new_device/authorize is used with two arguments:

• token is the new device token.
  • If new device token is incorrect, does not exist of expired, 404 error is returned
• device is the device name.
  • If the name already taken, random suffix will be added.
  • Name SHOULD only contain [^a-zA-Z0-9]. All other symbols will be replaced with _

sequenceDiagram
 actor A as Authorized device
 participant Server
 actor B as New device
 A->>+Server: /auth/new_device
 Server-->>A: Mnemonic token
 Note over A,B: Mnemonic token is used by new device to get access token
 B->>Server: /auth/new_device/authorize
 Server-->>-B: Access token

Recovery token

Recovery tokens are 24 random bytes encoded to the mnemonic phrase. Optionally, token lifetime and allowed uses can be limited.

Only one recovery token can exist at a time. If a new token is generated, the old one is deleted.

POST /auth/recovery_token is used to generate a recovery token. There are two optional arguments:

• expiration is the datetime when token must expire.
  • MUST follow the %Y-%m-%dT%H:%M:%S.%fZ format.
  • Returns 400 if time format is incorrect
  • Returns 400 if date is in the past
• uses is how many times the recovery token can be used
  • MUST be 1 or more.

POST /auth/recovery_token/use to use the token. Two required arguments:

• token is the recovery token
  • If recovery token is incorrect, does not exist of expired, 404 error is returned
• device is the device name
  • If the name already taken, random suffix will be added.
  • Name SHOULD only contain [^a-zA-Z0-9]. All other symbols will be replaced with _

Migrations

Migrations module is introduced in [[changelog#1 1 1|v1.1.1]] of API and provides one-shot migrations which cannot be performed from the NixOS configuration file changes. These migrations are checked and ran before every start of the API.

You can disable certain migrations if needed by creating an array at api.skippedMigrations in userdata.json and populating it with IDs of the migrations to skip. Adding DISABLE_ALL to that array disables the migrations module entirely.

For example:

...
    "api": {
        "token": "secret_token",
        "enableSwagger": false,
		"skippedMigrations": [ "fix_nixos_config_branch", "create_tokens_json" ]
    },
...

Fix NixOS configuration branch

Migration ID: fix_nixos_config_branch

Introduced in: [[changelog#1 1 1|v1.1.1]]

Description

Mobile SelfPrivacy app introduced a bug in version 0.4.0. New servers were initialized with a rolling-testing nixos config branch. This was fixed in app version 0.4.2, but existing servers were not updated. This migration fixes this by changing the nixos config branch to master.

Run conditions

Git repository at /etc/nixos is using rolling-testing branch.

Migration process

1. Moving context to /etc/nixos.
2. Running git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/* to fix the consequences of --single-branch flag.
3. git fetch --all to fetch all existing branches.
4. git pull to pull them.
5. git checkout master moves to master.
6. Leaving the context.

Create tokens JSON file

Migration ID: create_tokens_json

Introduced in: [[changelog#1 2 0|v1.2.0]]

Description

Selfprivacy API used a single token in userdata.json for authentication. This migration creates a new tokens.json file with the old token in it.

Run conditions

/etc/nixos/userdata/tokens.json does not exist.

Migration process

1. Current token is retrieved from userdata.json.
2. tokens.json is created with 0600 permission.
3. File is populated with the following data:

{
	"tokens": [
		{
			"token": "token_string",
			"name": "primary_token",
			"date": "current date from str(datetime.now())",
		}
	]
}

Testing

We use pytest. coverage run -m pytest && coverage xml && coverage report

Code formatting

Black.

Security

Bandit.

Linting

Pylint is recommended, but not enforced right now.

SPCVE-0001

API versions affected: [[changelog#Input sanitization added swagger https git selfprivacy org SelfPrivacy selfprivacy-rest-api pulls 5|All pre-1.1.0 releases]]

SelfPrivacy app versions affected: ≤0.2.4; fixed in 0.3.0

Discovered on: 16 Nov 2021

Addressed on: 17 Nov 2021

Description

Remote code execution vulnerability allowed root access to anyone, without any authorization. Was caused by the following factors:

• API had no authentication.
• No input sanitation used.
• Python's subprocess.Popen was called with shell=True.

At that time, there was no mechanism to upgrade API, so the server had to be recreated.

Taken measures

• Basic API auth added.
• All subprocess calls now don't use shell=true.
• CI pipeline now includes bandit to prevent same mistakes in the future.
• More input sanitation added.
• Created a nix overlay to provide API upgrades automatically.

selfprivacy-nixos-infect

Drone configuration

Drone pipeline

Stage Default:

It is ensured that teardown step must happen whether the pipeline fails or succeeds, because we need the server to be clean before every new job. It is done by forcing the step to be called with status trigger:

- name: teardown
  . . . 
  when:
    status:
    - failure
    - success

Reproted problems

NixOS fails to upgrade

We have to deploy our own channel with verified builds.

Volumes are not used properly

We need a controller which will detect them, format them, expand them and also move apps between them. Probably app should use Hetzner API to expand volume automatically.

Server control is lost when losing app's storage

Solution implemented on API side as recovery tokens in [[changelog#authorization_tokens branch|1.2.0]]. We need to implement this on app side.

Ability to change user passwords

Has to be implemented on both server and app side.

Server restoration from backup

There are several problems

We don't backup current userdata from nixos-config

So settings and users are not restored.

App don't check for the current users list

But we already have an endpoint for that.

Services are not stopped during restoration

Leads to files corruption?

User is not notified that they may have to add a credit card to b2

Free limits are pretty small and restoration may easily spend them all, and after that, everything is broken.

Disk storage is not monitored

Which leads to big problems for the server. API has to control available space, and how each app uses it. App should alert user and give options on storage management.

Backlog

No check for invalid DNSSEC setup

If DNSSEC is set up incorrectly, app won't tell user why server can't be set up.

Email can only use [a-z]

Some users wanted emails in format name.surname@domain.tld, but couldn't. As we create unix users, we have to use r"^[a-z_][a-z0-9_]+$". User documentation needs explanations on this?

Problems with international domains

I can't determine the scope of the problems it causes, without having one. But it surely creates a lot of problems on many stages, starting at server creation. Should we buy an international domain?

VPN

Still looking for another protocol.

HTTP headers on nextcloud are not optimal

X-Content-Type-Options is not set to "nosniff" and X-Frame-Options not set to "SAMEORIGIN".

Need more info and logs on... everything?

Especially on server creation. If we could display real-time logs of server setup, it would be perfect.

Free TLDs don't work

Reported as #84.

Servers are only deployed in Germany

Probably we should allow choosing the datacenter for the server. Also, in the future, provide more service providers.

ACME sometimes fails

We should at least start by exposing logs.

Some users want an in-depth video

I think we should make it after we handle major features of our upcoming release.

Roadmap

ToDo:

• Restore access to server
• Update control
• Disk autoresize
• SSH key add
• Fuzzing B)
• Password symbols (base64?)
• DKIM
• https://status.selfprivacy.org
• Jitsi
• OpenConnect

Desktop Release

• Linux
• Mac
• Windows

Second 100 users

• User docs

Google Playmarket, iOS AppStore

• Paid support system
• Law
• Check requrements

Backlog

• SP Infra monitoring (goss, squadcast)