⚠️ This guide is not officially supported by the Dify Team.⚠️ This is a community-edited, simplified version of the official migration guide presented by the Dify Team.
Complete guide to safely migrate Dify knowledge bases from Weaviate 1.19 to 1.27/1.33.
If your environment contains only a small number of Knowledges, you might be able to resolve the issue using the following much simpler steps, instead of the more complicated process on this page.
- Open the
Settingspage for your knowledge. - Change the
Embedding Modelto something else. - On the
Documentspage, wait until all documents becomeAvailable. - Open the
Settingspage again and change theEmbedding Modelback to the original. - On the
Documentspage, wait again until all documents becomeAvailable. - Repeat these steps for each Knowledges.
The steps described in the following sections are aimed at large environments, where it's not feasible to manually edit every Knowledges.
This guide covers the following two cases.
While Case A is recommended for a safer migration, this guide can also be applied to Case B:
- Case A
- You are currently running a version of Dify 1.9.1 or earlier with Weaviate 1.19 included.
- All knowledge is functioning properly.
- Case B
- You have already upgraded to Weaviate 1.27+ and are running Dify 1.9.2 or later.
- The knowledge created with the previous version is corrupted, and you have no backup to revert to the earlier version.
The procedure in this guide is as follows:
- Take a complete backup of your current Dify environment.
- If your Dify version is 1.9.1 or earlier, upgrade Dify.
- Operate the weaviate container and modify the directory structure of the LSM data.
- Operate the worker container and run the migration script.
- Perform cleanup.
Note:
This procedure cannot be rolled back by any means other than a restore. Attempting to roll back using anything other than a restore may make things worse.
We recommend that you follow the steps to take a full backup first, in preparation for a possible restore.
Stop your Dify services:
cd /path/to/dify/docker
docker compose downThen making full copy or archive of your entire docker directory (/path/to/dify/docker for example) as a safety measure.
If you encounter issues later, you can restore this backup to revert to the original state.
This step is only for Case A - users currently on Dify 1.9.1 or earlier with Weaviate 1.19.
If you are already running Weaviate 1.27+ (Case B), you can skip this step.
Follow the upgrade guide to move to the latest (or a specific) Dify version that uses Weaviate 1.27+.
- Upgrade guide: https://github.com/langgenius/dify/releases
If your Dify has stopped, start it and wait until it has fully launched.
cd /path/to/dify/docker
docker compose up -dEnsure your Weaviate using the image version 1.27.0 or higher.
cd /path/to/dify/docker
docker compose ps weaviate # The "IMAGE" column should show "semitechnologies/weaviate:1.27.0" or higherEnter the shell of your weaviatwe container:
cd /path/to/dify/docker
docker compose exec -it weaviate /bin/shThen run the following commands inside the container to fix LSM data:
cd /var/lib/weaviate
for dir in vector_index_*_node_*_lsm; do
[ -d "$dir" ] || continue
# Extract index ID and shard ID
index_id=$(echo "$dir" | sed -n 's/vector_index_\([^_]*_[^_]*_[^_]*_[^_]*_[^_]*\)_node_.*/\1/p')
shard_id=$(echo "$dir" | sed -n 's/.*_node_\([^_]*\)_lsm/\1/p')
# Create target directory and copy
mkdir -p "vector_index_${index_id}_node/$shard_id/lsm"
cp -a "$dir/"* "vector_index_${index_id}_node/$shard_id/lsm/"
echo "✓ Copied $dir"
done
exitThen restart weaviate container to ensure changes are recognized:
cd /path/to/dify/docker
docker compose restart weaviatePlace migrate_weaviate_collections.py script to your /path/to/dify/docker/volumes/app/storage/ directory, then enter the shell of your worker container:
cp /path/to/migrate_weaviate_collections.py /path/to/dify/docker/volumes/app/storage/
cd /path/to/dify/docker
docker compose exec -it worker /bin/bashThen run the following commands inside the container to execute the migration script:
uv run --no-cache /app/api/storage/migrate_weaviate_collections.py
exitRestart Dify services:
docker compose down
docker compose up -dVerify in Dify UI:
- Go to your Dify console
- Open your knowledge bases
- Try "Retrieval Testing"
- Should work without errors!
After successful migration, you can delete orphaned files to free up space.
Enter the shell of your weaviatwe container:
cd /path/to/dify/docker
docker compose exec -it weaviate /bin/shThen run the following commands inside the container to delete orphaned files:
cd /var/lib/weaviate
rm -rf vector_index_*_node_*
exitAlso, you can delete the migration script from your storage volume:
rm /path/to/dify/docker/volumes/app/storage/migrate_weaviate_collections.pymigrate_weaviate_collections.py- Schema migration script
- Original migration approach: Dify team
- LSM recovery method: Chinese Dify community user
- Combined solution: Community effort
@kurokobo
Unfortunately, the migration has not been successful.
It seems some trial and error may be required.
Due to the gRPC migration, it appears that after switching Weaviate to gRPC, the
index_node_idreturned from the vector database is now being treated as a UUID object rather than a string.This has caused SQLAlchemy to generate UUID-type parameters, and PostgreSQL is failing to perform the type conversion.
Consequently, the following error was being output:
Therefore, during the migration process, explicit modifications should be made to return UUID values as strings instead. While I plan to try implementing these corrections, I wanted to note that this approach appears unworkable in its current state, so I'm leaving this comment for reference.