Key rate-limit system thanks to Leander, full proxy list support

.gitignore

@@ -1,7 +1,20 @@
-*.zip
+# Environments
+# !!! KEEP THESE ENTRIES AT THE TOP OF THIS FILE BECAUSE THEY CONTAIN THE MOST SENSITIVE DATA !!!
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+env.old/
+.prod.env
+# !!! KEEP THESE ENTRIES AT THE TOP OF THIS FILE BECAUSE THEY CONTAIN THE MOST SENSITIVE DATA !!!
 
+rate_limited_keys.json
 last_update.txt
 
+*.zip
 *.log.json
 /logs
 /log
@@ -142,15 +155,6 @@ celerybeat.pid
 # SageMath parsed files
 *.sage.py
 
-# Environments
-.env
-.venv
-env/
-venv/
-ENV/
-env.bak/
-venv.bak/
-
 # Spyder project settings
 .spyderproject
 .spyproject
@@ -184,3 +188,4 @@ cython_debug/
 
 backups/
 cache/
+api/cache/rate_limited_keys.json

.vscode/settings.json

@@ -7,6 +7,7 @@
         "**/.DS_Store": true,
         "**/Thumbs.db": true,
         "**/__pycache__": true,
+        "**/*.css.map": true,
         "**/.vscode": true,
         "**/*.map": true,
         "tests/__pycache__": true

PUSH_TO_PRODUCTION.sh

@@ -22,4 +22,4 @@ cp env/.prod.env /home/nova-prod/.env
 cd /home/nova-prod
 
 # Start screen
-screen -S nova-api python run prod && sleep 5
+screen -L -S nova-api python run prod && sleep 5

README.md

@@ -125,14 +125,45 @@ Set up a MongoDB database and set `MONGO_URI` to the MongoDB database connection
 Want to use a proxy list? See the according section!
 Keep in mind to set `USE_PROXY_LIST` to `True`! Otherwise, the proxy list won't be used.
 
-### `ACTUAL_IPS` (optional)
+### Proxy Lists
+To use proxy lists, navigate to `api/secret/proxies/` and create the following files:
+- `http.txt`
+- `socks4.txt`
+- `socks5.txt`
+
+Then, paste your proxies in the following format:
+
+```
+[username:password@]host:port
+```
+Whereas anything inside of `[]` is optional and the host can be an IP address or a hostname. Always specify the port.
+
+If you're using [iproyal.com](https://iproyal.com?r=307932)<sup>affiliate link</sup>, follow the following steps:
+- Order any type of proxy or proxies
+- In the *Product Info* tab:
+  - set *Select port* to `SOCKS5`
+  - and *Select format* to `USER:PASS@IP:PORT`
+
+#### Proxy List Examples
+```
+1.2.3.4:8080
+user:pass@127.0.0.1:1337
+aaaaaaaaaaaaa:bbbbbbbbbb@1.2.3.4:5555
+```
+
+In the proxy credential files, can use comments just like in Python.
+
+**Important:** to activate the proxy lists, you need to change the `USE_PROXY_LIST` environment variable to `True`!
+
+
+### ~~`ACTUAL_IPS` (optional)~~ (deprecated, might come back in the future)
 This is a security measure to make sure a proxy, VPN, Tor or any other IP hiding service is used by the host when accessing "Closed"AI's API.
 It is a space separated list of IP addresses that are allowed to access the API.
 You can also just add the *beginning* of an API address, like `12.123.` (without an asterisk!) to allow all IPs starting with `12.123.`.
 > To disable the warning if you don't have this feature enabled, set `ACTUAL_IPS` to `None`.
 
 ### Timeout
-`TRANSFER_TIMEOUT` seconds to wait until the program throws an exception for if the request takes too long. We recommend rather long times like `120` for two minutes.
+`TRANSFER_TIMEOUT` seconds to wait until the program throws an exception for if the request takes too long. We recommend rather long times like `500` for 500 seconds.
 
 ### Core Keys
 `CORE_API_KEY` specifies the **very secret key** for  which need to access the entire user database etc.
@@ -145,29 +176,6 @@ You can also just add the *beginning* of an API address, like `12.123.` (without
 ### Other
 `KEYGEN_INFIX` can be almost any string (avoid spaces or special characters) - this string will be put in the middle of every NovaAI API key which is generated. This is useful for identifying the source of the key using e.g. RegEx.
 
-## Proxy Lists
-To use proxy lists, navigate to `api/secret/proxies/` and create the following files:
-- `http.txt`
-- `socks4.txt`
-- `socks5.txt`
-
-Then, paste your proxies in the following format:
-
-```
-[username:password@]host:port
-```
-
-e.g.
-
-```
-1.2.3.4:8080
-user:pass@127.0.0.1:1337
-```
-
-You can use comments just like in Python.
-
-**Important:** to use the proxy lists, you need to change the `USE_PROXY_LIST` environment variable to `True`!
-
 ## Run
 > **Warning:** read the according section for production usage!
 
@@ -186,17 +194,20 @@ python run 1337
 ```
 
 ## Adding a provider
+To be documented!]
 
-## Test if it works
+## Run tests
+Make sure the API server is running on the port you specified and run:
 `python checks`
 
-## Ports
+## Default Ports
 ```yml
-2332: Developement (default)
+2332: Developement
 2333: Production
 ```
 
 ## Production
+
 Make sure your server is secure and up to date.
 Check everything.
 

api/cache/crypto_prices.json

@@ -1 +0,0 @@
-{"LTC": 64.665, "_last_updated": 1695334741.4905503, "BTC": 26583.485, "MATIC": 0.52075, "XMR": 146.46058828041404, "ADA": 0.2455, "USDT": 1.000005, "ETH": 1586.115, "USD": 1.0, "EUR": 1.0662838016640013}

api/cache/rate_limited_keys.json

@@ -1 +0,0 @@
-[]

api/db/key_validation.py

@@ -7,8 +7,8 @@ from dotenv import load_dotenv
 from motor.motor_asyncio import AsyncIOMotorClient
 
 load_dotenv()
-MONGO_URI = os.getenv("MONGO_URI")
 
+MONGO_URI = os.getenv('MONGO_URI')
 
 async def log_rated_key(key: str) -> None:
     """Logs a key that has been rate limited to the database."""
@@ -16,8 +16,8 @@ async def log_rated_key(key: str) -> None:
     client = AsyncIOMotorClient(MONGO_URI)
 
     scheme = {
-        "key": key,
+        'key': key,
-        "timestamp_added": int(time.time())
+        'timestamp_added': int(time.time())
     }
 
     collection = client['Liabilities']['rate-limited-keys']
@@ -31,7 +31,7 @@ async def key_is_rated(key: str) -> bool:
     collection = client['Liabilities']['rate-limited-keys']
 
     query = {
-        "key": key
+        'key': key
     }
 
     result = await collection.find_one(query)
@@ -39,9 +39,9 @@ async def key_is_rated(key: str) -> bool:
 
 
 async def cached_key_is_rated(key: str) -> bool:
-    path = os.path.join(os.getcwd(), "cache", "rate_limited_keys.json")
+    path = os.path.join(os.getcwd(), 'cache', 'rate_limited_keys.json')
 
-    with open(path, "r") as file:
+    with open(path, 'r') as file:
         keys = json.load(file)
 
     return key in keys
@@ -62,14 +62,13 @@ async def remove_rated_keys() -> None:
             marked_for_removal.append(key['_id'])
 
     query = {
-        "_id": {
+        '_id': {
-            "$in": marked_for_removal
+            '$in': marked_for_removal
         }
     }
 
     await collection.delete_many(query)
 
-
 async def cache_all_keys() -> None:
     """Clones all keys from the database to the cache."""
 
@@ -79,8 +78,8 @@ async def cache_all_keys() -> None:
     keys = await collection.find().to_list(length=None)
     keys = [key['key'] for key in keys]
 
-    path = os.path.join(os.getcwd(), "cache", "rate_limited_keys.json")
+    path = os.path.join(os.getcwd(), 'cache', 'rate_limited_keys.json')
-    with open(path, "w") as file:
+    with open(path, 'w') as file:
         json.dump(keys, file)
 
 if __name__ == "__main__":

api/main.py

@@ -16,6 +16,8 @@ from slowapi import Limiter, _rate_limit_exceeded_handler
 
 from helpers import network
 
+# 
+
 import core
 import handler
 
@@ -37,7 +39,7 @@ limiter = Limiter(
     swallow_errors=True,
     key_func=get_remote_address,
     default_limits=[
-    '2/second',
+    '1/second',
     '20/minute',
     '300/hour'
 ])
@@ -71,8 +73,8 @@ async def v1_handler(request: fastapi.Request):
     res = await handler.handle(incoming_request=request)
     return res
 
-@limiter.limit('100/second')
+@limiter.limit('100/minute', '1000/hour') 
-@app.route('/enterprise/{path:path}', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH'])
+@app.route('/enterprise/v1/{path:path}', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH'])
 async def enterprise_handler(request: fastapi.Request):
     res = await handler.handle(incoming_request=request)
     return res

api/proxies.py

@@ -45,7 +45,10 @@ class Proxy:
 
         self.proxy_type = proxy_type
         self.host_or_ip = host_or_ip
-        self.ip_address = socket.gethostbyname(self.host_or_ip) # get ip address from host
+        try:
+            self.ip_address = socket.gethostbyname(self.host_or_ip) # get ip address from host
+        except socket.gaierror:
+            self.ip_address = self.host_or_ip
         self.host = self.host_or_ip
         self.port = port
         self.username = username
@@ -61,6 +64,17 @@ class Proxy:
         self.urls_httpx = {k + '://' :v for k, v in self.urls.items()}
         self.proxies = self.url
 
+        print({
+            'proxy_type': self.proxy_type,
+            'host_or_ip': self.host_or_ip,
+            'ip_address': self.ip_address,
+            'host': self.host,
+            'port': self.port,
+            'username': self.username,
+            'password': self.password,
+            'url': self.url
+        })
+
     @property
     def connector(self):
         """
@@ -78,7 +92,7 @@ class Proxy:
 
         return aiohttp_socks.ProxyConnector(
             proxy_type=proxy_types[self.proxy_type],
-            host=self.ip_address,
+            host=self.host,
             port=self.port,
             rdns=False, 
             username=self.username,
@@ -89,16 +103,15 @@ class Proxy:
 
 proxies_in_files = []
 
-try:
+for proxy_type in ['http', 'socks4', 'socks5']:
-    for proxy_type in ['http', 'socks4', 'socks5']:
+    try:
         with open(f'secret/proxies/{proxy_type}.txt') as f:
             for line in f:
                 clean_line = line.split('#', 1)[0].strip()
                 if clean_line:
                     proxies_in_files.append(f'{proxy_type}://{clean_line}')
-
+    except FileNotFoundError:
-except FileNotFoundError:
+        pass
-    pass
 
 ## Manages the proxy list
 
@@ -115,6 +128,8 @@ def get_proxy() -> Proxy:
     The proxy is either from the proxy list or from the environment variables.
     """
 
+    print('URL:\t' + ProxyLists().get_random.url)
+
     if USE_PROXY_LIST:
         return ProxyLists().get_random
 

api/responder.py

@@ -46,7 +46,7 @@ async def respond(
         'Content-Type': 'application/json'
     }
 
-    for i in range(20):
+    for _ in range(20):
         # Load balancing: randomly selecting a suitable provider
         # If the request is a chat completion, then we need to load balance between chat providers
         # If the request is an organic request, then we need to load balance between organic providers
@@ -86,7 +86,7 @@ async def respond(
                     cookies=target_request.get('cookies'),
                     ssl=False,
                     timeout=aiohttp.ClientTimeout(
-                        connect=0.5,
+                        connect=0.3,
                         total=float(os.getenv('TRANSFER_TIMEOUT', '500'))
                     ),
                 ) as response:
@@ -104,14 +104,16 @@ async def respond(
                                 pass
 
                             case _:
+                                key = target_request.get('provider_auth')
+
                                 match error.get('code'):
-                                    case "insufficient_quota":
+                                    case 'invalid_api_key':
-                                        key = target_request.get('provider_auth')
                                         await key_validation.log_rated_key(key)
-                                        continue
+                                        print('[!] invalid key', key)
+                                        pass
 
                                     case _:
-                                        pass
+                                        print('[!] unknown error with key: ', key, error)
 
                         if 'method_not_supported' in str(data):
                             await errors.error(500, 'Sorry, this endpoint does not support this method.', data['error']['message'])
@@ -124,6 +126,10 @@ async def respond(
                         if response.ok:
                             json_response = data
 
+                        else:
+                            continue
+
+
                     if is_stream:
                         try:
                             response.raise_for_status()
@@ -139,6 +145,7 @@ async def respond(
                     break
 
             except Exception as exc:
+                print('[!] exception', exc)
                 continue
 
             if (not json_response) and is_chat:
@@ -146,7 +153,7 @@ async def respond(
                 continue
     else:
         print('[!] no response')
-        yield await errors.yield_error(500, 'Sorry, the provider is not responding.', 'Please try again later.')
+        yield await errors.yield_error(500, 'Sorry, our API seems to have issues connecting to our provider(s).', 'This most likely isn\'t your fault. Please try again later.')
         return
 
     if (not is_stream) and json_response: