Galileo AI: Complete Guide & Tutorial

Introduction to Galileo AI

Building applications with Large Language Models (LLMs) is exciting, but it comes with unique challenges. Unlike traditional software, where you can trace a bug to a specific line of code, LLM outputs can be unpredictable, inconsistent, or even hallucinated. How do you know if your AI application is performing well in production? How do you catch a model that suddenly starts generating incorrect answers? This is where Galileo AI comes in.

Galileo AI is an observability and evaluation platform specifically designed for teams working with LLMs. Think of it as a diagnostic dashboard for your AI application. It provides real-time monitoring, prompt evaluation, and detailed analytics to help you understand exactly what your LLM is doing, why it is doing it, and how to make it better. Whether you are building a customer support chatbot, a code generator, or a content summarizer, Galileo AI gives you the tools to ensure your AI is reliable, accurate, and safe.

This tutorial will walk you through everything you need to know to get started with Galileo AI, from initial setup to advanced debugging techniques. By the end, you will be able to monitor your AI applications with confidence and fix issues before they affect your users.

Getting Started with Galileo AI

Before you can start monitoring your LLM, you need to set up your Galileo AI account and connect it to your application. The process is straightforward and designed to integrate with your existing workflow.

Step 1: Create an Account

Navigate to https://galileo.ai/ and sign up for a free account. Galileo offers a generous free tier that allows you to monitor a limited number of requests per month, which is perfect for experimentation and small projects. Once you sign up, you will be taken to your project dashboard.

Step 2: Create a New Project

Inside the dashboard, click on the “Create Project” button. You will be asked to give your project a name (e.g., “Customer Support Bot” or “Content Generator”). You will also be prompted to select the LLM framework you are using. Galileo supports major frameworks including:

• OpenAI (GPT-3.5, GPT-4, GPT-4o)
• LangChain (popular for building chains and agents)
• LlamaIndex (great for RAG applications)
• Hugging Face models
• Anthropic Claude

Select your framework to get a tailored setup guide.

Step 3: Install the SDK

Galileo provides SDKs for Python and JavaScript/TypeScript. For this tutorial, we will focus on Python, which is the most common language for LLM development. Open your terminal and install the Galileo package using pip:

pip install galileo-sdk

Once installed, you need to initialize the SDK with your API key. You can find your API key in the Galileo dashboard under “Settings” > “API Keys”. In your code, add the following lines at the very beginning of your application:

import galileo as g
g.init(api_key="YOUR_GALILEO_API_KEY")

This single line of code enables monitoring for all LLM calls made within your application.

Step 4: Instrument Your LLM Calls

Galileo works by wrapping your LLM calls. If you are using OpenAI directly, you can use the Galileo wrapper. For example:

from galileo import wrap_openai
import openai

openai = wrap_openai(openai)

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain quantum computing in simple terms."}]
)

If you are using LangChain, Galileo provides a callback handler that automatically captures all traces. Simply add it to your chain:

from galileo.integrations.langchain import GalileoCallbackHandler

galileo_handler = GalileoCallbackHandler()
chain = LLMChain(llm=llm, prompt=prompt, callbacks=[galileo_handler])

Once these steps are complete, every LLM request your application makes will be automatically logged and analyzed in your Galileo dashboard.

Key Features of Galileo AI

Galileo AI is packed with features that help you understand and improve your LLM application. Let’s break down the most important ones.

Real-Time LLM Observability and Monitoring

This is the core feature of Galileo. As your application runs, Galileo captures every request, response, token usage, and latency metric. You can see live data streaming into your dashboard. This is incredibly useful for spotting anomalies immediately. For example, if your response times suddenly spike, or if the model starts returning empty responses, you will see it in real-time.

Prompt Evaluation and Testing

One of the biggest challenges in LLM development is prompt engineering. A small change in wording can dramatically change the output. Galileo allows you to run batch evaluations on your prompts. You can define test cases with expected outputs, and Galileo will compare the model’s actual responses against your expectations. This feature is essential for regression testing when you update your prompts or switch models.

Hallucination Detection and Debugging

Hallucinations—where the model confidently generates false information—are a major risk in production. Galileo uses advanced algorithms to detect potential hallucinations in your outputs. When a hallucination is flagged, you can drill down into the specific request, see the exact prompt, the model’s response, and the context provided. This makes debugging much faster than trying to reproduce the issue manually.

Performance Analytics and Dashboards

Galileo provides pre-built dashboards that visualize key metrics such as:

• Token usage – How many tokens are being consumed per request and per user.
• Latency – How long each request takes, broken down by model and endpoint.
• Error rates – How often the model returns errors or empty responses.
• Cost tracking – Estimate your API costs based on actual usage.

You can customize these dashboards to focus on the metrics that matter most to your team.

Integration with Popular LLM Frameworks

Galileo is designed to work seamlessly with the tools you already use. Beyond LangChain and OpenAI, it supports LlamaIndex, Chroma, Pinecone, and many others. This means you can add observability to your existing stack without rewriting your code.

How to Use Galileo AI: A Practical Walkthrough

Let’s walk through a common use case: building a customer support chatbot and using Galileo to monitor and improve it.

Step 1: Set Up Monitoring

Assume you have a simple chatbot that uses a RAG (Retrieval-Augmented Generation) pipeline. You have a vector database of your company’s documentation, and you retrieve relevant chunks before sending them to the LLM. After instrumenting your code as described in the Getting Started section, run your chatbot for a few hours. Go to the Galileo dashboard and click on your project. You should see a live feed of requests.

Step 2: Analyze a Single Request

Click on any request in the feed. You will see a detailed trace showing:

• The exact prompt sent to the LLM (including the retrieved documents).
• The full response from the LLM.
• Token count and latency.
• A score for context relevance (how well the retrieved documents match the query).
• A score for answer faithfulness (whether the response is supported by the provided context).

If you see a low faithfulness score, that is a red flag for potential hallucination.

Step 3: Create a Test Suite for Prompt Evaluation

You want to ensure your chatbot always responds politely and never makes up information. Go to the “Evaluation” tab in Galileo. Create a new test suite. Add test cases like:

• Query: “What is your return policy?” Expected: A response that mentions the 30-day return window.
• Query: “Can I get a refund?” Expected: A response that does not promise a refund without conditions.

Run the test suite against your current prompt. Galileo will execute each test case and show you a pass/fail status. If a test fails, you can see exactly what the model output was and compare it to your expectation. Adjust your prompt and rerun the tests until all pass.

Step 4: Detect and Debug a Hallucination

Imagine a user asks, “Does your product support integration with Salesforce?” Your chatbot responds with, “Yes, we have a native Salesforce integration.” However, your company does not actually have this integration. This is a hallucination. In Galileo, filter your requests by low faithfulness scores. You will find this specific request. Click on it. You can see that the retrieved documents did not mention Salesforce at all. The model made it up. Now you can either add a document about integrations to your vector database, or modify your system prompt to say, “Only answer based on the provided context.”

Step 5: Set Up Alerts

You cannot watch the dashboard 24/7. Galileo allows you to set up alerts. For example, you can create an alert that sends an email or a Slack message when the hallucination rate exceeds 5% in the last hour. Go to “Alerts” in the dashboard, choose your metric (e.g., faithfulness score), set the threshold, and choose the notification channel. Now you will be notified immediately when something goes wrong.

Tips for Getting the Most Out of Galileo AI

Here are some practical tips that will help you use Galileo AI more effectively, especially if you are new to LLM observability.

Start with a Single Use Case

Do not try to monitor every part of your application at once. Pick one specific flow—for example, the onboarding chatbot or the FAQ generator—and instrument that first. Once you are comfortable with the data and dashboards, expand to other parts of your system.

Use Custom Metrics

Galileo allows you to define custom metrics. For instance, if you are building a code generation tool, you might want to track how often the generated code compiles. You can send this data to Galileo and see it alongside the standard metrics. This gives you a holistic view of your application’s quality.

Version Your Prompts

When you run evaluations in Galileo, you can tag your prompts with version numbers. This is extremely helpful when you are iterating on prompt engineering. You can compare version 1.0 of your prompt against version 2.0 and see which one produces better results across your test suite. This data-driven approach removes guesswork from prompt design.

Monitor the Context Quality

If you are using RAG, pay close attention to the context relevance score. A low score often indicates that your retrieval pipeline is returning irrelevant documents. This is a common source of poor answers. Use Galileo to identify which queries have low context relevance, then improve your retrieval logic or your document chunking strategy.

Integrate with Your CI/CD Pipeline

For production-grade applications, consider integrating Galileo evaluations into your continuous integration pipeline. Before you deploy a new prompt or a new model, run your test suite automatically. If the pass rate drops below a threshold, block the deployment. This prevents regressions from reaching your users.

Use the “Compare” Feature

When you are testing different models (e.g., GPT-3.5 vs. GPT-4) or different prompt strategies, use Galileo’s compare feature. You can run the same set of requests through two different configurations and see side-by-side metrics for latency, cost, and quality. This makes it easy to justify decisions to your team.

Do Not Ignore Latency Spikes

Sometimes a sudden increase in latency is not a network issue—it can be a sign that your prompt is too long or that the model is struggling with a complex instruction. Galileo’s latency breakdown shows you exactly where the time is spent. Use this data to optimize your prompts and reduce costs.

Conclusion

Galileo AI is an essential tool for any team serious about building reliable LLM applications. It transforms the “black box” of AI outputs into a transparent, analyzable system. By providing real-time monitoring, prompt evaluation, and hallucination detection, it gives you the confidence to deploy AI in production without fear of unexpected failures. Start small, use the test suites, and let the data guide your improvements. Your users—and your team—will thank you.