> ## Documentation Index
> Fetch the complete documentation index at: https://portkey-docs-add-third-party-integration-issues-fixes.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# vLLM
> Integrate vLLM-hosted custom models with Portkey for production observability and reliability.
Portkey provides a robust platform to observe, govern, and manage your **locally** or **privately** hosted custom models using vLLM.
Here's a [list](https://docs.vllm.ai/en/latest/models/supported_models.html) of all model architectures supported on vLLM.
## Integration Steps
Expose your vLLM server using a tunneling service like [ngrok](https://ngrok.com/) or make it publicly accessible. Skip this if you're self-hosting the Gateway.
```sh theme={null}
ngrok http 8000 --host-header="localhost:8080"
```
1. Go to [**Model Catalog → Add Provider**](https://app.portkey.ai/model-catalog/providers)
2. Enable **"Local/Privately hosted provider"** toggle
3. Select **OpenAI** as the provider type (vLLM follows OpenAI API schema)
4. Enter your vLLM server URL in **Custom Host**: `https://your-vllm-server.ngrok-free.app`
5. Add authentication headers if needed
6. Name your provider (e.g., `my-vllm`)
See all setup options
```python Python theme={null}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@my-vllm"
)
response = portkey.chat.completions.create(
model="your-model-name",
messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
```
```javascript Node.js theme={null}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: '@my-vllm'
});
const response = await portkey.chat.completions.create({
model: 'your-model-name',
messages: [{ role: 'user', content: 'Hello!' }]
});
console.log(response.choices[0].message.content);
```
**Or use custom host directly:**
```python Python theme={null}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="openai",
custom_host="https://your-vllm-server.ngrok-free.app",
Authorization="AUTH_KEY" # If needed
)
```
```javascript Node.js theme={null}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: 'openai',
customHost: 'https://your-vllm-server.ngrok-free.app',
Authorization: 'AUTH_KEY' // If needed
});
```
**Important:** vLLM follows the OpenAI API specification, so set the provider as `openai` when using custom host directly. By default, vLLM runs on `http://localhost:8000/v1`.
***
## Next Steps
Add retries, timeouts, and fallbacks
Monitor your vLLM deployments
Learn more about custom host setup
Complete guide for private LLMs
For complete SDK documentation:
Complete Portkey SDK documentation