Getting Started with Haystack¶

Search is a topic of ever increasing importance. Users increasingly rely on search to separate signal from noise and find what they’re looking for quickly. In addition, search can provide insight into what things are popular (many searches), what things are difficult to find on the site and ways you can improve the site.

To this end, Haystack tries to make integrating custom search as easy as possible while being flexible/powerful enough to handle more advanced use cases.

Haystack is a reusable app (that is, it relies only on its own code and focuses on providing just search) that plays nicely with both apps you control as well as third-party apps (such as django.contrib.*) without having to modify the sources.

Haystack also does pluggable backends (much like Django’s database layer), so virtually all of the code you write ought to be portable between whichever search engine you choose.

Note

If you hit a stumbling block, there is both a mailing list and #haystack on irc.freenode.net to get help.

Note

You can participate in and/or track the development of Haystack by subscribing to the development mailing list.

This tutorial assumes that you have a basic familiarity with the various major parts of Django (models/forms/views/settings/URLconfs) and tailored to the typical use case. There are shortcuts available as well as hooks for much more advanced setups, but those will not be covered here.

For example purposes, we’ll be adding search functionality to a simple note-taking application. Here is myapp/models.py:

from django.db import models
from django.contrib.auth.models import User


class Note(models.Model):
    user = models.ForeignKey(User)
    pub_date = models.DateTimeField()
    title = models.CharField(max_length=200)
    body = models.TextField()

    def __unicode__(self):
        return self.title

Finally, before starting with Haystack, you will want to choose a search backend to get started. There is a quick-start guide to Installing Search Engines, though you may want to defer to each engine’s official instructions.

Installation¶

Use your favorite Python package manager to install the app from PyPI, e.g.

Example:

pip install django-haystack

When using elasticsearch, use:

pip install "django-haystack[elasticsearch]"

Configuration¶

Add Haystack To `INSTALLED_APPS`¶

As with most Django applications, you should add Haystack to the INSTALLED_APPS within your settings file (usually settings.py).

Example:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',

    # Added.
    'haystack',

    # Then your usual apps...
    'blog',
]

Modify Your `settings.py`¶

Within your settings.py, you’ll need to add a setting to indicate where your site configuration file will live and which backend to use, as well as other settings for that backend.

HAYSTACK_CONNECTIONS is a required setting and should be at least one of the following:

Solr¶

Example (Solr 4.X):

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr'
        # ...or for multicore...
        # 'URL': 'http://127.0.0.1:8983/solr/mysite',
    },
}

Example (Solr 6.X):

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr/tester',                 # Assuming you created a core named 'tester' as described in installing search engines.
        'ADMIN_URL': 'http://127.0.0.1:8983/solr/admin/cores'
        # ...or for multicore...
        # 'URL': 'http://127.0.0.1:8983/solr/mysite',
    },
}

Elasticsearch¶

Example (ElasticSearch 1.x):

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },
}

Example (ElasticSearch 2.x):

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch2_backend.Elasticsearch2SearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },
}

Example (ElasticSearch 5.x):

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch5_backend.Elasticsearch5SearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },
}

Example (ElasticSearch 7.x):

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch7_backend.Elasticsearch7SearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },
}

Whoosh¶

Requires setting PATH to the place on your filesystem where the Whoosh index should be located. Standard warnings about permissions and keeping it out of a place your webserver may serve documents out of apply.

Example:

import os
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
        'PATH': os.path.join(os.path.dirname(__file__), 'whoosh_index'),
    },
}

Xapian¶

First, install the Xapian backend (via http://github.com/notanumber/xapian-haystack/tree/master) per the instructions included with the backend.

Requires setting PATH to the place on your filesystem where the Xapian index should be located. Standard warnings about permissions and keeping it out of a place your webserver may serve documents out of apply.

Example:

import os
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': os.path.join(os.path.dirname(__file__), 'xapian_index'),
    },
}

Simple¶

The simple backend using very basic matching via the database itself. It’s not recommended for production use but it will return results.

Warning

This backend does NOT work like the other backends do. Data preparation does nothing & advanced filtering calls do not work. You really probably don’t want this unless you’re in an environment where you just want to silence Haystack.

Example:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.simple_backend.SimpleEngine',
    },
}

Handling Data¶

Creating `SearchIndexes`¶

SearchIndex objects are the way Haystack determines what data should be placed in the search index and handles the flow of data in. You can think of them as being similar to Django Models or Forms in that they are field-based and manipulate/store data.

You generally create a unique SearchIndex for each type of Model you wish to index, though you can reuse the same SearchIndex between different models if you take care in doing so and your field names are very standardized.

To build a SearchIndex, all that’s necessary is to subclass both indexes.SearchIndex & indexes.Indexable, define the fields you want to store data with and define a get_model method.

We’ll create the following NoteIndex to correspond to our Note model. This code generally goes in a search_indexes.py file within the app it applies to, though that is not required. This allows Haystack to automatically pick it up. The NoteIndex should look like:

import datetime
from haystack import indexes
from myapp.models import Note


class NoteIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    author = indexes.CharField(model_attr='user')
    pub_date = indexes.DateTimeField(model_attr='pub_date')

    def get_model(self):
        return Note

    def index_queryset(self, using=None):
        """Used when the entire index for model is updated."""
        return self.get_model().objects.filter(pub_date__lte=datetime.datetime.now())

Every SearchIndex requires there be one (and only one) field with document=True. This indicates to both Haystack and the search engine about which field is the primary field for searching within.

Warning

When you choose a document=True field, it should be consistently named across all of your SearchIndex classes to avoid confusing the backend. The convention is to name this field text.

There is nothing special about the text field name used in all of the examples. It could be anything; you could call it pink_polka_dot and it won’t matter. It’s simply a convention to call it text.

To use a document field with a name other than text, be sure to configure the HAYSTACK_DOCUMENT_FIELD setting. For example,:

HAYSTACK_DOCUMENT_FIELD = 'pink_polka_dot'

Additionally, we’re providing use_template=True on the text field. This allows us to use a data template (rather than error-prone concatenation) to build the document the search engine will index. You’ll need to create a new template inside your template directory called search/indexes/myapp/note_text.txt and place the following inside:

{{ object.title }}
{{ object.user.get_full_name }}
{{ object.body }}

In addition, we added several other fields (author and pub_date). These are useful when you want to provide additional filtering options. Haystack comes with a variety of SearchField classes to handle most types of data.

A common theme is to allow admin users to add future content but have it not display on the site until that future date is reached. We specify a custom index_queryset method to prevent those future items from being indexed.

Setting Up The Views¶

Add The `SearchView` To Your URLconf¶

Within your URLconf, add the following line:

path('search/', include('haystack.urls')),

This will pull in the default URLconf for Haystack. It consists of a single URLconf that points to a SearchView instance. You can change this class’s behavior by passing it any of several keyword arguments or override it entirely with your own view.

Search Template¶

Your search template (search/search.html for the default case) will likely be very simple. The following is enough to get going (your template/block names will likely differ):

{% extends 'base.html' %}

{% block content %}
    <h2>Search</h2>

    <form method="get" action=".">
        <table>
            {{ form.as_table }}
            <tr>
                <td>&nbsp;</td>
                <td>
                    <input type="submit" value="Search">
                </td>
            </tr>
        </table>

        {% if query %}
            <h3>Results</h3>

            {% for result in page.object_list %}
                <p>
                    <a href="{{ result.object.get_absolute_url }}">{{ result.object.title }}</a>
                </p>
            {% empty %}
                <p>No results found.</p>
            {% endfor %}

            {% if page.has_previous or page.has_next %}
                <div>
                    {% if page.has_previous %}<a href="?q={{ query }}&amp;page={{ page.previous_page_number }}">{% endif %}&laquo; Previous{% if page.has_previous %}</a>{% endif %}
                    |
                    {% if page.has_next %}<a href="?q={{ query }}&amp;page={{ page.next_page_number }}">{% endif %}Next &raquo;{% if page.has_next %}</a>{% endif %}
                </div>
            {% endif %}
        {% else %}
            {# Show some example queries to run, maybe query syntax, something else? #}
        {% endif %}
    </form>
{% endblock %}

Note that the page.object_list is actually a list of SearchResult objects instead of individual models. These objects have all the data returned from that record within the search index as well as score. They can also directly access the model for the result via {{ result.object }}. So the {{ result.object.title }} uses the actual Note object in the database and accesses its title field.

Reindex¶

The final step, now that you have everything setup, is to put your data in from your database into the search index. Haystack ships with a management command to make this process easy.

Note

If you’re using the Solr backend, you have an extra step. Solr’s configuration is XML-based, so you’ll need to manually regenerate the schema. You should run ./manage.py build_solr_schema first, drop the XML output in your Solr’s schema.xml file and restart your Solr server.

Simply run ./manage.py rebuild_index. You’ll get some totals of how many models were processed and placed in the index.

Note

Using the standard SearchIndex, your search index content is only updated whenever you run either ./manage.py update_index or start afresh with ./manage.py rebuild_index.

You should cron up a ./manage.py update_index job at whatever interval works best for your site (using --age=<num_hours> reduces the number of things to update).

Alternatively, if you have low traffic and/or your search engine can handle it, the RealtimeSignalProcessor automatically handles updates/deletes for you.

Complete!¶

You can now visit the search section of your site, enter a search query and receive search results back for the query! Congratulations!

What’s Next?¶

This tutorial just scratches the surface of what Haystack provides. The SearchQuerySet is the underpinning of all search in Haystack and provides a powerful, QuerySet-like API (see SearchQuerySet API). You can use much more complicated SearchForms/SearchViews to give users a better UI (see Views & Forms). And the Best Practices provides insight into non-obvious or advanced usages of Haystack.

Getting Started with Haystack¶

Installation¶

Configuration¶

Add Haystack To `INSTALLED_APPS`¶

Modify Your `settings.py`¶

Solr¶

Elasticsearch¶

Whoosh¶

Xapian¶

Simple¶

Handling Data¶

Creating `SearchIndexes`¶

Setting Up The Views¶

Add The `SearchView` To Your URLconf¶

Search Template¶

Reindex¶

Complete!¶

What’s Next?¶

Haystack

Navigation

Related Topics

Getting Started with Haystack¶

Installation¶

Configuration¶

Add Haystack To INSTALLED_APPS¶

Modify Your settings.py¶

Solr¶

Elasticsearch¶

Whoosh¶

Xapian¶

Simple¶

Handling Data¶

Creating SearchIndexes¶

Setting Up The Views¶

Add The SearchView To Your URLconf¶

Search Template¶

Reindex¶

Complete!¶

What’s Next?¶

Add Haystack To `INSTALLED_APPS`¶

Modify Your `settings.py`¶

Creating `SearchIndexes`¶

Add The `SearchView` To Your URLconf¶