📌 How to use another LLM other than OpenAI or Local LLM

(how to configure Yaml) ?

AutoRAG can use all LLMs supported by the LlamaIndex

1. Use vllm module

We recommend using VLLM for fast inference!

We developed it parallelize, so you can experiment faster than using the llama_index_llm module :)

• Docs: vllm

• Sample YAML

  modules:
  	-module_type: vllm
  	 llm: mistralai/Mistral-7B-Instruct-v0.2
  	 temperature:[0.1, 1.0]
  	 max_tokens: 512
  

2. Use llama_index_llm module

• Docs: llama_index_llm supporting llm models

• Sample YAML

  modules:
    - module_type: llama_index_llm
      llm: openailike
      model: mistralai/Mistral-7B-Instruct-v0.2
      api_base: your_api_base
      api_key: your_api_key
  

❗ How to use LLMs other than the 3 LLM Model Types (e.g. ollama, groq)

[Tutorial] Use Ollama

1. Add ollama with the code below in python

import autorag
from llama_index.llms.ollama import Ollama

autorag.generator_models["ollama"] = Ollama

1. Configuring a YAML file

nodes:
  - node_line_name: node_line_1
    nodes:
      - node_type: generator
        modules:
          - module_type: llama_index_llm
            llm: ollama
            model: [llama3, qwen, mistral]

Additional parameters can be used directly in a YAML file