We are excited to announce a new feature for our CRM customers: Custom AI API Endpoints. This feature allows you to use your own AI models or open source alternatives to the ones provided by OpenAI, depending on your needs and preferences.
Custom Endpoints can help you save money on AI costs while enhancing your data privacy. You can run your own AI models on your own hardware, without sending any data to third-party services. For example, you can use the open source Whisper model to transcribe call recordings using the same API as OpenAI. Whisper is a large language model that can run on a consumer GPU with 11GB of RAM.
Large language models are powerful AI tools that can generate natural language texts for various purposes. They are trained on massive amounts of data and learn to capture the patterns and nuances of human language. As the size of these models increases, they can exhibit emergent capabilities that go beyond their original training objectives. For example, some large language models can answer questions, write summaries, generate code, and even create music or art.
However, running large language models requires a lot of computational resources, especially video RAM (VRAM). VRAM is the memory that is used by the GPU to process graphics and other tasks. The more VRAM a GPU has, the larger and more complex models it can run. For example, a 30 billion parameter model requires 24GB of VRAM, which is available in some high-end consumer GPUs. As of June 2023, recent models of this size are comparable to GPT-3.5, one of the most advanced models from OpenAI. By combining multiple GPUs, it is possible to run even larger models with more emergent capabilities.
Open source large language models are advancing at a rapid rate, thanks to the efforts of researchers and developers around the world. As our customer, you have the option of using OpenAI’s models or hosting your own open source models. As hardware capability inevitably improves and prices come down, you will be able to run more powerful and privately hosted models that can easily integrate with your existing customer data.
The future is bright and full of possibilities as we leverage these technologies in service of society. If you’re interested in exploring this option, please contact us for more details. We hope you enjoy this new feature and we look forward to hearing your feedback.