The Hugging Face Concepts Library and Importing Textual Inversion files#
Using Textual Inversion Files#
Textual inversion (TI) files are small models that customize the output of Stable Diffusion image generation. They can augment SD with specialized subjects and artistic styles. They are also known as "embeds" in the machine learning world.
Each TI file introduces one or more vocabulary terms to the SD model. These are
known in InvokeAI as "triggers." Triggers are often, but not always, denoted
using angle brackets as in "<trigger-phrase>". The two most common type of
TI files that you'll encounter are .pt and .bin files, which are produced by
different TI training packages. InvokeAI supports both formats, but its
built-in TI training system produces .pt.
The Hugging Face company has amassed a large ligrary of >800 community-contributed TI files covering a broad range of subjects and styles. InvokeAI has built-in support for this library which downloads and merges TI files automatically upon request. You can also install your own or others' TI files by placing them in a designated directory.
An Example#
Here are a few examples to illustrate how it works. All these images were generated using the command-line client and the Stable Diffusion 1.5 model:
| Japanese gardener | Japanese gardener <ghibli-face> | Japanese gardener <hoi4-leaders> | Japanese gardener <cartoona-animals> |
|---|---|---|---|
 |
 |
 |
 |
You can also combine styles and concepts:
| A portrait of <alf> in <cartoona-animal> style |
|---|
 |
Using a Hugging Face Concept#
Authenticating to HuggingFace
Some concepts require valid authentication to HuggingFace. Without it, they will not be downloaded and will be silently ignored.
If you used an installer to install InvokeAI, you may have already set a HuggingFace token. If you skipped this step, you can:
- run the InvokeAI configuration script again (if you used a manual installer):
invokeai-configure - set one of the
HUGGINGFACE_TOKENorHUGGING_FACE_HUB_TOKENenvironment variables to contain your token
Finally, if you already used any HuggingFace library on your computer, you might already have a token
in your local cache. Check for a hidden .huggingface directory in your home folder. If it
contains a token file, then you are all set.
Hugging Face TI concepts are downloaded and installed automatically as you require them. This requires your machine to be connected to the Internet. To find out what each concept is for, you can browse the Hugging Face concepts library and look at examples of what each concept produces.
When you have an idea of a concept you wish to try, go to the command-line
client (CLI) and type a < character and the beginning of the Hugging Face
concept name you wish to load. Press Tab, and the CLI will show you all
matching concepts. You can also type < and hit Tab to get a listing of all
~800 concepts, but be prepared to scroll up to see them all! If there is more
than one match you can continue to type and Tab until the concept is
completed.
Example
if you type in <x and hit Tab, you'll be prompted with the completions:
Now type id and press Tab. It will be autocompleted to <xidiversity>
because this is a unique match.
Finish your prompt and generate as usual. You may include multiple concept terms in the prompt.
If you have never used this concept before, you will see a message that the TI
model is being downloaded and installed. After this, the concept will be saved
locally (in the models/sd-concepts-library directory) for future use.
Several steps happen during downloading and installation, including a scan of
the file for malicious code. Should any errors occur, you will be warned and the
concept will fail to load. Generation will then continue treating the trigger
term as a normal string of characters (e.g. as literal <ghibli-face>).
You can also use <concept-names> in the WebGUI's prompt textbox. There is no
autocompletion at this time.
Installing your Own TI Files#
You may install any number of .pt and .bin files simply by copying them into
the embeddings directory of the InvokeAI runtime directory (usually invokeai
in your home directory). You may create subdirectories in order to organize the
files in any way you wish. Be careful not to overwrite one file with another.
For example, TI files generated by the Hugging Face toolkit share the named
learned_embedding.bin. You can use subdirectories to keep them distinct.
At startup time, InvokeAI will scan the embeddings directory and load any TI
files it finds there. At startup you will see a message similar to this one:
Note the * trigger term. This is a placeholder term that many early TI
tutorials taught people to use rather than a more descriptive term.
Unfortunately, if you have multiple TI files that all use this term, only the
first one loaded will be triggered by use of the term.
To avoid this problem, you can use the merge_embeddings.py script to merge two
or more TI files together. If it encounters a collision of terms, the script
will prompt you to select new terms that do not collide. See
Textual Inversion for details.
Further Reading#
Please see the repository and associated paper for details and limitations.
Created: November 30, 2022