Documented almost everything

This commit is contained in:
nsde 2023-10-12 00:03:15 +02:00
parent 169f5469a9
commit eb6768cae5
27 changed files with 183 additions and 269 deletions

139
README.md
View file

@ -74,6 +74,20 @@ This one's code can be found in the following repository: [github.com/novaoss/no
- `screen` (for production)
- Cloudflare (for security, anti-DDoS, etc.) - we fully support Cloudflare
## Staging System
This repository has an integrated staging system. It's a simple system that allows you to test the API server before deploying it to production.
You should definitely set up two databases on MongoDB: `nova-core` and `nova-test`. Please note that `nova-core` is always used for `providerkeys`.
Put your production `.env` file in `env/.prod.env`. Your test `.env` file should be in `.env`.
Running `PUSH_TO_PRODUCTION.sh` will:
- kill port `2333` (production)
- remove all contents of the production directory, set to `/home/nova-prod/` (feel free to change it)
- then copy the test directory (generally *this* directory) to the production directory
- copy the `.env` file from `env/.prod.env` to `.env`
- use `screen` to run the production server on port `2333`
## Install
Assuming you have a new version of Python 3.9+ and pip installed:
```py
@ -108,14 +122,13 @@ pip install alt-profanity-check
pip install git+https://github.com/dimitrismistriotis/alt-profanity-check.git
```
## `.env` configuration
Create a `.env` file, make sure not to reveal any of its contents to anyone, and fill in the required values in the format `KEY=VALUE`. Otherwise, the code won't run.
### Database
Set up a MongoDB database and set `MONGO_URI` to the MongoDB database connection URI. Quotation marks are definetly recommended here!
### Proxy
### Proxy (optional)
- `PROXY_TYPE` (optional, defaults to `socks.PROXY_TYPE_HTTP`): the type of proxy - can be `http`, `https`, `socks4`, `socks5`, `4` or `5`, etc...
- `PROXY_HOST`: the proxy host (host domain or IP address), without port!
- `PROXY_PORT` (optional)
@ -125,7 +138,7 @@ Set up a MongoDB database and set `MONGO_URI` to the MongoDB database connection
Want to use a proxy list? See the according section!
Keep in mind to set `USE_PROXY_LIST` to `True`! Otherwise, the proxy list won't be used.
### Proxy Lists
### Proxy Lists (optional)
To use proxy lists, navigate to `api/secret/proxies/` and create the following files:
- `http.txt`
- `socks4.txt`
@ -176,7 +189,99 @@ You can also just add the *beginning* of an API address, like `12.123.` (without
### Other
`KEYGEN_INFIX` can be almost any string (avoid spaces or special characters) - this string will be put in the middle of every NovaAI API key which is generated. This is useful for identifying the source of the key using e.g. RegEx.
## Run
## Misc
`api/cache/models.json` has to be a valid JSON file in the OpenAI format. It is what `/v1/models` always returns. Make sure to update it regularly.
Example: https://pastebin.com/raw/WuNzTJDr (updated ~aug/2023)
## Providers
This is one of the most essential parts of NovaAI. Providers are the APIs used to access the AI models.
The modules are located in `api/providers/` and active providers are specified in `api/providers/__init__.py`.
You shouldn't use `.env` for provider keys. Instead, use the database as it's more flexible and secure. For a detailed explanation, scroll down a bit to the database section.
Always return the `provider_auth` dictionary key if you a API key is required. Example:
```py
...
return {
...
'provider_auth': f'exampleprovider>{key}'
}
...
```
Whereas `exampleprovider` is the provider name used in the database.
### Check providers
List all providers using `python api`. This **won't** start the API server.
Check if a provider is working using `python api <provider>`, e.g. `python api azure`. This **doesn't** require the API to be running.
## Core Database
You need to set up a MongoDB database and set `MONGO_URI` in `.env` to the MongoDB database connection URI. Use quotation marks!
It's also important to set up the database `nova-core`.
The following collections are used in the database and will be created automatically if they don't exist.
### `users`
Generally, the `api_key` should be treated as the primary key. However, the `discord` and `github` keys are also unique and can be used as primary keys.
- `api_key`: API key [str]
- `credits`: credits [int]
- `role` (optional): credit multiplier. Check `api/config/config.yml`. [str]
- `status`:
- `active`: defaults to `true`. May be used in the future so that users can deactivate their accounts.
- `ban_reason`: defaults to `""`. If the user is banned, this will be set to the reason.
- `auth`:
- `discord`: Discord user ID. Use a string, because Discord IDs can be larger than the maximum integer value.
- `github`: GitHub user ID. Not used yet.
- `level`: Discord (Arcane bot) level. [int]
### `providerkeys`
Used in `api/providers/...` to store API keys of the providers.
- `provider`: provider name [str], e.g. `azure``
- `key`: API key [str], e.g. `sk-...`
- `rate_limited_since`: timestamp [int], defaults to `null`. The unix timestamp when the provider was rate limited. If it's `null`, the provider is not rate limited.
- `inactive_reason`: defaults to `null`. If the key is disabled or terminated, this will be set to the reason automatically. You can also set it manually. [str]
- `source`: just to keep track of where the key came from. [str]
### `stats`
Logs general statistics.
Automatically updated by the API server.
More info is yet to be documented.
### `logs`
Every API request is logged here.
Automatically updated by the API server.
More info is yet to be documented.
## Finance Database
The finance database is used to keep track of the finances.
**Important:** *always* use the specified `currency` for the field `amount`.
### `donations`
- `currency`: (crypto) currency [str], e.g. EUR, USD, BTC, USDT-TRX, ...
- `amount`: amount [float].
- `user`: [str], Discord user ID
- `proof`: [str], link to proof (e.g. transaction hash)
- `timestamp`: [int], unix timestamp
### `expenses`
- `currency`: (crypto) currency [str], e.g. EUR, USD, BTC, USDT-ETH, ...
- `amount`: amount [float]. NOT negative!
- `proof`: [str], link to proof (e.g. transaction hash)
- `type`: [str], type of expense, e.g. `wage`, `payment`, `donation`, ...
- `to`: [str], entity the expense was paid to, e.g. `IPRoyal`, `employee/John`
- `reason`: [str], reason for the expense
- `timestamp`: [int], unix timestamp
## Run Test Server
> **Warning:** read the according section for production usage!
For developement:
@ -192,33 +297,19 @@ You can also specify a port, e.g.:
```bash
python run 1337
```
## Adding a provider
To be documented!]
## Run tests
## Tests
Make sure the API server is running on the port you specified and run:
`python checks`
## Default Ports
It is recommended to use the default ports, because this will make it easier to set other parts of the infrastructure up.
```yml
2332: Developement
2333: Production
```
## Production
## Run Production Server
Make sure your server is secure and up to date.
Check everything.
The following command will run the API __without__ a reloader!
```bash
python run prod
```
or
```bash
./screen.sh
```
Make sure you have read all the according sections and have set up everything correctly.

View file

@ -1,2 +0,0 @@
# import pruner
# pruner.prune()

View file

@ -1,86 +0,0 @@
# import os
# import sys
# project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
# sys.path.append(project_root)
# from api.db.users import UserManager
# manager = UserManager()
# async def update_credits(settings=None):
# users = await manager.get_all_users()
# if not settings:
# await users.update_many({}, {'$inc': {'credits': 2500}})
# else:
# for key, value in settings.items():
# await users.update_many(
# {'level': key}, {'$inc': {'credits': int(value)}})
# get_all_users = manager.get_all_users
# ###
# import os
# import time
# import aiohttp
# import pymongo
# import asyncio
# import autocredits
# from settings import roles
# from dotenv import load_dotenv
# load_dotenv()
# async def main():
# await update_roles()
# await autocredits.update_credits(roles)
# async def update_roles():
# async with aiohttp.ClientSession() as session:
# try:
# async with session.get('http://0.0.0.0:3224/user_ids') as response:
# discord_users = await response.json()
# except aiohttp.ClientError as e:
# print(f'Error: {e}')
# return
# level_role_names = [f'lvl{lvl}' for lvl in range(10, 110, 10)]
# users_doc = await autocredits.get_all_users()
# users = users_doc.find({})
# users = await users.to_list(length=None)
# for user in users:
# if not 'auth' in user:
# continue
# discord = str(user['auth']['discord'])
# for user_id, role_names in discord_users.items():
# if user_id == discord:
# for role in level_role_names:
# if role in role_names:
# users_doc.update_one(
# {'auth.discord': discord},
# {'$set': {'level': role}}
# )
# print(f'Updated {discord} to {role}')
# return users
# def launch():
# asyncio.run(main())
# with open('rewards/last_update.txt', 'w', encoding='utf8') as f:
# f.write(str(time.time()))
# # ====================================================================================
# if __name__ == '__main__':
# launch()

View file

@ -11,6 +11,8 @@ async def after_request(
is_chat: bool,
model: str,
) -> None:
"""Runs after every request."""
if user and incoming_request:
await logs.log_api_request(user=user, incoming_request=incoming_request, target_url=target_request['url'])
@ -20,7 +22,7 @@ async def after_request(
ip_address = await network.get_ip(incoming_request)
await stats.manager.add_date()
await stats.manager.add_ip_address(ip_address)
# await stats.manager.add_ip_address(ip_address)
await stats.manager.add_path(path)
await stats.manager.add_target(target_request['url'])

View file

@ -1,8 +1,6 @@
import os
import json
import asyncio
import aiofiles
import aiofiles.os
from sys import argv
from bson import json_util
@ -14,13 +12,14 @@ load_dotenv()
MONGO_URI = os.environ['MONGO_URI']
FILE_DIR = os.path.dirname(os.path.realpath(__file__))
async def main(output_dir: str):
async def main(output_dir: str) -> None:
await make_backup(output_dir)
async def make_backup(output_dir: str):
async def make_backup(output_dir: str) -> None:
"""Makes a backup of all collections in the database."""
output_dir = os.path.join(FILE_DIR, '..', 'backups', output_dir)
await aiofiles.os.makedirs(output_dir, exist_ok=True)
os.makedirs(output_dir, exist_ok=True)
client = AsyncIOMotorClient(MONGO_URI)
databases = await client.list_database_names()
@ -30,22 +29,24 @@ async def make_backup(output_dir: str):
if database == 'local':
continue
await aiofiles.os.makedirs(os.path.join(output_dir, database), exist_ok=True)
os.makedirs(os.path.join(output_dir, database), exist_ok=True)
for collection in databases[database]:
print(f'Initiated database backup for {database}/{collection}')
print(f'Initiated backup for {database}/{collection}')
await make_backup_for_collection(database, collection, output_dir)
print(f'Finished backup for {database}/{collection}')
async def make_backup_for_collection(database, collection, output_dir):
async def make_backup_for_collection(database, collection, output_dir) -> None:
"""Makes a backup of a collection in the database."""
path = os.path.join(output_dir, database, f'{collection}.json')
client = AsyncIOMotorClient(MONGO_URI)
collection = client[database][collection]
documents = await collection.find({}).to_list(length=None)
async with aiofiles.open(path, 'w') as f:
with open(path, 'w') as f:
for chunk in json.JSONEncoder(default=json_util.default).iterencode(documents):
await f.write(chunk)
f.write(chunk)
if __name__ == '__main__':
if len(argv) < 2 or len(argv) > 2:

View file

@ -1,11 +1,10 @@
max-credits: 100001
max-credits-owner: 694201337
start-credits: 1000
start-credits: 1000 # Credits given to new users.
# Credit cost per API request.
costs:
other: 5
other: 5 # Other endpoints
chat-models:
chat-models: # chat completions
gpt-4-32k: 200
gpt-4: 50
gpt-3: 10

View file

@ -14,7 +14,6 @@ import hmac
import httpx
import fastapi
import aiofiles
import functools
from dhooks import Webhook, Embed
from dotenv import load_dotenv
@ -163,6 +162,7 @@ async def get_crypto_price(cryptocurrency: str) -> float:
if is_old or cryptocurrency not in cache:
async with httpx.AsyncClient() as client:
response = await client.get(f'https://api.coinbase.com/v2/prices/{cryptocurrency}-USD/spot')
response.raise_for_status()
usd_price = float(response.json()['data']['amount'])
cache[cryptocurrency] = usd_price

View file

@ -13,7 +13,9 @@ class FinanceManager:
async def _get_collection(self, collection_name: str):
return self.conn['finances'][collection_name]
async def get_entire_financial_history(self):
async def get_entire_financial_history(self) -> dict:
"""Returns the entire financial history of the organization."""
donations_db = await self._get_collection('donations')
expenses_db = await self._get_collection('expenses')

View file

@ -19,14 +19,9 @@ conn = AsyncIOMotorClient(os.environ['MONGO_URI'])
async def _get_collection(collection_name: str):
return conn[os.getenv('MONGO_NAME', 'nova-test')][collection_name]
async def replacer(text: str, dict_: dict) -> str:
# This seems to exist for a very specific and dumb purpose :D
for k, v in dict_.items():
text = text.replace(k, v)
return text
async def log_api_request(user: dict, incoming_request, target_url: str):
"""Logs the API Request into the database."""
db = await _get_collection('logs')
payload = {}

View file

@ -3,9 +3,6 @@ import time
import random
import asyncio
import aiofiles
import aiofiles.os
from aiocache import cached
from dotenv import load_dotenv
from cachetools import TTLCache
from motor.motor_asyncio import AsyncIOMotorClient
@ -22,6 +19,8 @@ class KeyManager:
return self.conn['nova-core'][collection_name]
async def add_key(self, provider: str, key: str, source: str='?'):
"""Adds a key to the database."""
db = await self._get_collection('providerkeys')
await db.insert_one({
'provider': provider,
@ -32,6 +31,8 @@ class KeyManager:
})
async def get_possible_keys(self, provider: str):
"""Returns a list of possible keys for a provider."""
db = await self._get_collection('providerkeys')
keys = await db.find({
'provider': provider,
@ -45,6 +46,8 @@ class KeyManager:
return keys
async def get_key(self, provider: str):
"""Returns a random key for a provider."""
keys = await self.get_possible_keys(provider)
if not keys:
@ -55,6 +58,8 @@ class KeyManager:
return api_key
async def rate_limit_key(self, provider: str, key: str, duration: int):
"""Rate limits a key for a provider."""
db = await self._get_collection('providerkeys')
await db.update_one({'provider': provider, 'key': key}, {
'$set': {
@ -63,6 +68,8 @@ class KeyManager:
})
async def deactivate_key(self, provider: str, key: str, reason: str):
"""Deactivates a key for a provider, and gives a reason."""
db = await self._get_collection('providerkeys')
await db.update_one({'provider': provider, 'key': key}, {
'$set': {
@ -71,12 +78,14 @@ class KeyManager:
})
async def import_all(self):
"""Imports all keys from the secret/ folder."""
db = await self._get_collection('providerkeys')
num = 0
for filename in await aiofiles.os.listdir(os.path.join('api', 'secret')):
for filename in os.listdir(os.path.join('api', 'secret')):
if filename.endswith('.txt'):
async with aiofiles.open(os.path.join('api', 'secret', filename)) as f:
async open(os.path.join('api', 'secret', filename)) as f:
async for line in f:
if not line.strip():
continue

View file

@ -4,7 +4,6 @@ import os
import json
import yaml
import time
import orjson
import fastapi
from dotenv import load_dotenv

View file

@ -12,12 +12,7 @@ async def get_ip(request) -> str:
if request.headers.get('x-forwarded-for'):
xff, *_ = request.headers['x-forwarded-for'].split(', ')
possible_ips = [
xff,
request.headers.get('cf-connecting-ip'),
request.client.host
]
possible_ips = [xff, request.headers.get('cf-connecting-ip'), request.client.host]
detected_ip = next((i for i in possible_ips if i), None)
return detected_ip

View file

@ -11,7 +11,7 @@ async def _get_module_name(module) -> str:
async def balance_chat_request(payload: dict) -> dict:
"""
### Load balance the chat completion request between chat providers.
Load balance the chat completion request between chat providers.
Providers are sorted by streaming and models. Target (provider.chat_completion) is returned
"""
@ -38,7 +38,7 @@ async def balance_chat_request(payload: dict) -> dict:
async def balance_organic_request(request: dict) -> dict:
"""
### Load balance non-chat completion request
Load balance non-chat completion request
Balances between other "organic" providers which respond in the desired format already.
Organic providers are used for non-chat completions, such as moderation and other paths.
"""

View file

@ -1,9 +1,9 @@
"""FastAPI setup."""
import os
import fastapi
import pydantic
from rich import print
from dotenv import load_dotenv
from bson.objectid import ObjectId
@ -50,11 +50,17 @@ async def startup_event():
# https://stackoverflow.com/a/74529009
pydantic.json.ENCODERS_BY_TYPE[ObjectId]=str
folders = ['backups', 'cache', 'secret', 'secret/proxies']
for folder in folders:
try:
os.mkdir(folder)
except FileExistsError:
pass
@app.get('/')
async def root():
"""
Returns general information about the API.
"""
"""Returns general information about the API."""
return {
'hi': 'Welcome to the Nova API!',

View file

@ -1,13 +1,10 @@
"""This module contains functions for checking if a message violates the moderation policy."""
import time
import difflib
import asyncio
import aiocache
import profanity_check
from typing import Union
from Levenshtein import distance
cache = aiocache.Cache(aiocache.SimpleMemoryCache)
@ -28,8 +25,7 @@ def input_to_text(inp: Union[str, list]) -> str:
return text
async def is_policy_violated(inp: Union[str, list]) -> bool:
"""Checks if the input violates the moderation policy.
"""
"""Checks if the input violates the moderation policy."""
# use aio cache to cache the result
inp = input_to_text(inp)

View file

@ -1,2 +1,2 @@
from . import azure, webraft
MODULES = [azure, webraft]
from . import azure
MODULES = [azure]

View file

@ -3,7 +3,6 @@ import sys
import aiohttp
import asyncio
import importlib
import aiofiles.os
from rich import print
@ -21,7 +20,7 @@ async def main():
except IndexError:
print('List of available providers:')
for file_name in await aiofiles.os.listdir(os.path.dirname(__file__)):
for file_name in os.listdir(os.path.dirname(__file__)):
if file_name.endswith('.py') and not file_name.startswith('_'):
print(file_name.split('.')[0])

View file

@ -1,10 +1,10 @@
from .helpers import utils
AUTH = True
ORGANIC = False
STREAMING = True
MODERATIONS = False
ENDPOINT = 'https://nova-00001.openai.azure.com'
AUTH = True # If the provider requires an API key
ORGANIC = False # If all OpenAI endpoints are available on the provider. If false, only a chat completions are available.
STREAMING = True # If the provider supports streaming completions
ENDPOINT = 'https://nova-00001.openai.azure.com' # (Important: read below) The endpoint for the provider.
#! IMPORTANT: If this is an ORGANIC provider, this should be the endpoint for the API with anything BEFORE the "/v1".
MODELS = [
'gpt-3.5-turbo',
'gpt-3.5-turbo-16k',

View file

@ -3,7 +3,6 @@ from .helpers import utils
AUTH = True
ORGANIC = True
STREAMING = True
MODERATIONS = True
ENDPOINT = 'https://api.openai.com'
MODELS = utils.GPT_3

View file

@ -3,7 +3,6 @@ from .helpers import utils
AUTH = True
ORGANIC = False
STREAMING = True
MODERATIONS = True
ENDPOINT = 'https://api.openai.com'
MODELS = utils.GPT_4

View file

@ -3,7 +3,6 @@ from .helpers import utils
AUTH = True
ORGANIC = False
STREAMING = True
MODERATIONS = False
ENDPOINT = 'https://api.openai.com'
MODELS = utils.GPT_4_32K

View file

@ -3,8 +3,6 @@
import os
import socket
import random
import asyncio
import aiohttp
import aiohttp_socks
from rich import print
@ -16,7 +14,7 @@ USE_PROXY_LIST = os.getenv('USE_PROXY_LIST', 'False').lower() == 'true'
class Proxy:
"""
### Represents a proxy.
Represents a proxy.
The type can be either http, https, socks4 or socks5.
You can also pass a url, which will be parsed into the other attributes.
@ -69,7 +67,7 @@ class Proxy:
@property
def connector(self):
"""
### Returns a proxy connector
Returns a proxy connector
Returns an aiohttp_socks.ProxyConnector object.
This can be used in aiohttp.ClientSession.
"""
@ -115,7 +113,7 @@ class ProxyLists:
def get_proxy() -> Proxy:
"""
### Returns a Proxy object
Returns a Proxy object
The proxy is either from the proxy list or from the environment variables.
"""

View file

@ -45,7 +45,8 @@ async def respond(
input_tokens: int=0,
incoming_request: starlette.requests.Request=None,
):
"""Stream the completions request. Sends data in chunks
"""
Stream the completions request. Sends data in chunks
If not streaming, it sends the result in its entirety.
"""

View file

@ -1,82 +0,0 @@
import os
import json
import openai
from dotenv import load_dotenv
load_dotenv()
openai.api_base = 'http://localhost:2332/v1'
openai.api_key = os.environ['NOVA_KEY']
# Example dummy function hard coded to return the same weather
# In production, this could be your backend API or an external API
def get_current_weather(location, unit='fahrenheit'):
"""Get the current weather in a given location"""
weather_info = {
'location': location,
'temperature': '72',
'unit': unit,
'forecast': ['sunny', 'windy'],
}
return json.dumps(weather_info)
def run_conversation():
# Step 1: send the conversation and available functions to GPT
messages = [{'role': 'user', 'content': 'What\'s the weather like in Boston?'}]
functions = [
{
'name': 'get_current_weather',
'description': 'Get the current weather in a given location',
'parameters': {
'type': 'object',
'properties': {
'location': {
'type': 'string',
'description': 'The city and state, e.g. San Francisco, CA',
},
'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit']},
},
'required': ['location'],
},
}
]
response = openai.ChatCompletion.create(
model='gpt-3.5-turbo-0613',
messages=messages,
functions=functions,
function_call='auto', # auto is default, but we'll be explicit
)
response_message = response['choices'][0]['message']
# Step 2: check if GPT wanted to call a function
if response_message.get('function_call'):
# Step 3: call the function
# Note: the JSON response may not always be valid; be sure to handle errors
available_functions = {
'get_current_weather': get_current_weather,
} # only one function in this example, but you can have multiple
function_name = response_message['function_call']['name']
fuction_to_call = available_functions[function_name]
function_args = json.loads(response_message['function_call']['arguments'])
function_response = fuction_to_call(
location=function_args.get('location'),
unit=function_args.get('unit'),
)
# Step 4: send the info on the function call and function response to GPT
messages.append(response_message) # extend conversation with assistant's reply
messages.append(
{
'role': 'function',
'name': function_name,
'content': function_response,
}
) # extend conversation with function response
second_response = openai.ChatCompletion.create(
model='gpt-3.5-turbo-0613',
messages=messages,
) # get a new response from GPT where it can see the function response
return second_response
print(run_conversation())

View file

@ -1,10 +0,0 @@
--- EXPECTED ---
data: {"id":"custom-chatcmpl-nUSiapqELukaPT7vEnGcXkbvrS1fR","object":"chat.completion.chunk","created":1696716717,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"id":"custom-chatcmpl-nUSiapqELukaPT7vEnGcXkbvrS1fR","object":"chat.completion.chunk","created":1696716717,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"123"},"finish_reason":null}]}
data: {"id":"custom-chatcmpl-nUSiapqELukaPT7vEnGcXkbvrS1fR","object":"chat.completion.chunk","created":1696716717,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

View file

@ -9,6 +9,8 @@ from api.db.users import UserManager
manager = UserManager()
async def update_credits(settings=None):
"""Updates the credits of all users."""
users = await manager.get_all_users()
if not settings:

View file

@ -15,6 +15,8 @@ async def main():
await autocredits.update_credits(roles)
async def update_roles():
"""Updates the roles of all users."""
async with aiohttp.ClientSession() as session:
try:
async with session.get('http://0.0.0.0:3224/get_roles') as response: