Dify: Weaviate 1.19 to 1.27+ Migration Guide (community-edited, simplified)

migrate_weaviate_collections_ce.py

"""
# NOTE: THIS SCRIPT IS DEPRECATED AND OUTDATED

## TL;DR;

Use the official migration script instead.
https://github.com/langgenius/dify-docs/blob/main/assets/migrate_weaviate_collections.py

## Background

This script was originally released as a community-edited version of the draft of the script presented by the Dify Team,
to address some issues encountered during migration of Weaviate collections in certain environments,
before the official script was finalized, as a temporary workaround.

However, all modifications made in this script have already been backported to the official script.
Therefore, this unofficial script is deprecated and will not be maintained in the future.
We strongly recommend using the latest official script.
If you face any issues with the official script, please report them to the Dify Team via their GitHub repository or any supported channels.

You can see the revisions made in this script by checking the git history: https://gist.github.com/kurokobo/51fbe7f92f4526957e12dacfa7783cdf/revisions
The original source for this script can be found at: https://github.com/langgenius/dify/issues/27291#issuecomment-3501003678.
The key changes made in this script were:

- Retrieve Weaviate connection info from environment variables to make this script run in the Worker container.
- Switch to cursor-based pagination in "replace_old_collection", since the migration could fail with large collections.
- Fix an issue where both the old and new collections remained without being deleted after migrating an empty collection.

"""

import sys

print("WARNING: This migration script is DEPRECATED and OUTDATED.")
print("Please use the following official migration script instead:")
print("https://github.com/langgenius/dify-docs/blob/main/assets/migrate_weaviate_collections.py")
print("This script will now exit without making any changes.")

sys.exit(1)

WEAVIATE_MIGRATION_GUIDE.md

Weaviate 1.19 to 1.27+ Migration Guide for Dify

• ⚠️ This guide is not officially supported by the Dify Team.
• ⚠️ This is a community-edited, simplified version of the official migration guide presented by the Dify Team.

Complete guide to safely migrate Dify knowledge bases from Weaviate 1.19 to 1.27/1.33.

---

✅ NOTE: BEFORE PROCEEDING FURTHER

If your environment contains only a small number of Knowledges, you might be able to resolve the issue using the following much simpler steps, instead of the more complicated process on this page.

1. Open the Settings page for your knowledge.
2. Change the Embedding Model to something else.
3. On the Documents page, wait until all documents become Available.
4. Open the Settings page again and change the Embedding Model back to the original.
5. On the Documents page, wait again until all documents become Available.
6. Repeat these steps for each Knowledges.

The steps described in the following sections are aimed at large environments, where it's not feasible to manually edit every Knowledges.

---

📝 Outline

This guide covers the following two cases.
While Case A is recommended for a safer migration, this guide can also be applied to Case B:

• Case A
  • You are currently running a version of Dify 1.9.1 or earlier with Weaviate 1.19 included.
  • All knowledge is functioning properly.
• Case B
  • You have already upgraded to Weaviate 1.27+ and are running Dify 1.9.2 or later.
  • The knowledge created with the previous version is corrupted, and you have no backup to revert to the earlier version.

The procedure in this guide is as follows:

1. Take a complete backup of your current Dify environment.
2. If your Dify version is 1.9.1 or earlier, upgrade Dify.
3. Operate the weaviate container and modify the directory structure of the LSM data.
4. Operate the worker container and run the migration script.
5. Perform cleanup.

---

📝 Migration Procedure

Note:
This procedure cannot be rolled back by any means other than a restore. Attempting to roll back using anything other than a restore may make things worse.
We recommend that you follow the steps to take a full backup first, in preparation for a possible restore.

---

Step 1: Backup Your Environment

Stop your Dify services:

cd /path/to/dify/docker
docker compose down

Then making full copy or archive of your entire docker directory (/path/to/dify/docker for example) as a safety measure.

If you encounter issues later, you can restore this backup to revert to the original state.

---

Step 2: Upgrade to Weaviate 1.27+ (Only for Case A)

This step is only for Case A - users currently on Dify 1.9.1 or earlier with Weaviate 1.19.
If you are already running Weaviate 1.27+ (Case B), you can skip this step.

Follow the upgrade guide to move to the latest (or a specific) Dify version that uses Weaviate 1.27+.

• Upgrade guide: https://github.com/langgenius/dify/releases

---

Step 3: Fix Orphaned LSM Data

If your Dify has stopped, start it and wait until it has fully launched.

cd /path/to/dify/docker
docker compose up -d

Ensure your Weaviate using the image version 1.27.0 or higher.

cd /path/to/dify/docker
docker compose ps weaviate  # The "IMAGE" column should show "semitechnologies/weaviate:1.27.0" or higher

Enter the shell of your weaviatwe container:

cd /path/to/dify/docker
docker compose exec -it weaviate /bin/sh

Then run the following commands inside the container to fix LSM data:

cd /var/lib/weaviate
for dir in vector_index_*_node_*_lsm; do
  [ -d "$dir" ] || continue
  
  # Extract index ID and shard ID
  index_id=$(echo "$dir" | sed -n 's/vector_index_\([^_]*_[^_]*_[^_]*_[^_]*_[^_]*\)_node_.*/\1/p')
  shard_id=$(echo "$dir" | sed -n 's/.*_node_\([^_]*\)_lsm/\1/p')
  
  # Create target directory and copy
  mkdir -p "vector_index_${index_id}_node/$shard_id/lsm"
  cp -a "$dir/"* "vector_index_${index_id}_node/$shard_id/lsm/"
  
  echo "✓ Copied $dir"
done
exit

Then restart weaviate container to ensure changes are recognized:

cd /path/to/dify/docker
docker compose restart weaviate

---

Step 4: Migrate Schema

Place migrate_weaviate_collections.py script to your /path/to/dify/docker/volumes/app/storage/ directory, then enter the shell of your worker container:

cp /path/to/migrate_weaviate_collections.py /path/to/dify/docker/volumes/app/storage/
cd /path/to/dify/docker
docker compose exec -it worker /bin/bash

Then run the following commands inside the container to execute the migration script:

uv run --no-cache /app/api/storage/migrate_weaviate_collections.py
exit

Restart Dify services:

docker compose down
docker compose up -d

Verify in Dify UI:

1. Go to your Dify console
2. Open your knowledge bases
3. Try "Retrieval Testing"
4. Should work without errors!

Step 5: Cleanup (Optional)

After successful migration, you can delete orphaned files to free up space.
Enter the shell of your weaviatwe container:

cd /path/to/dify/docker
docker compose exec -it weaviate /bin/sh

Then run the following commands inside the container to delete orphaned files:

cd /var/lib/weaviate
rm -rf vector_index_*_node_*
exit

Also, you can delete the migration script from your storage volume:

rm /path/to/dify/docker/volumes/app/storage/migrate_weaviate_collections.py

---

📝 Files Needed

• migrate_weaviate_collections.py - Schema migration script

📝 Credits

• Original migration approach: Dify team
• LSM recovery method: Chinese Dify community user
• Combined solution: Community effort

Some fail when index is too big:

Replacing old collection with migrated data...
  Step 1: Getting data from migrated collection...

Warning: Could not automatically replace collection: Query call with protocol GRPC search failed with message grpc: trying to send message larger than max (85746713 vs. 10485760).

To activate manually:
1. Delete the old collection: Vector_index_57db41fd_f350_407a_9203_44b73cbed92c_Node
2. Rename Vector_index_57db41fd_f350_407a_9203_44b73cbed92c_Node_migrated to Vector_index_57db41fd_f350_407a_9203_44b73cbed92c_Node

Here is the English translation:

---

My solution:

Update the environment variables of the Weaviate service in docker-compose.yml:

environment:
  - GRPC_MAX_MESSAGE_SIZE=1073741824  # 1GB
  - QUERY_DEFAULTS_LIMIT=100000       # Allow returning more data in a single query

Update the Weaviate connection parameters in the migrate_weaviate_collections.py script:

import weaviate
from weaviate.classes.init import AdditionalConfig, Timeout

client = weaviate.connect_to_local(
    host=WEAVIATE_HOST,
    port=WEAVIATE_PORT,
    grpc_port=WEAVIATE_GRPC_PORT,
    auth_credentials=weaviate.auth.AuthApiKey(WEAVIATE_API_KEY),
    additional_config=AdditionalConfig(
        timeout=Timeout(init=30, query=120, insert=240),
        grpc_max_message_length=1024 * 1024 * 1024,
    )
)

And increase the limit in replace_old_collection:

objects = migrated.query.fetch_objects(include_vector=True, limit=100000)

In my case, I had an Index handling up to 2,381,219 objects, and despite careful limit adjustments, I couldn't achieve optimal performance. Therefore, I reconfigured the system to repeatedly execute in batch sizes following the same approach used during migration.
Additionally, while GRPC_MAX_MESSAGE_SIZE theoretically has no upper limit, it appears to use an int32 data type, meaning approximately 2GB should be the maximum setting.

We had three indexes with roughly this maximum capacity, and the migration process completed in under three hours.
As with the original source, the ability to automatically check during initial processing whether migration was necessary was extremely helpful—rerunning the process allowed us to verify that no operations were missed.

Updated Script (Click to Expand)

#!/usr/bin/env python3
"""
Migration script to fix Weaviate schema incompatibility between 1.19.0 and 1.27.0+
 
This script:
- Identifies collections with old schema (no vectorConfig)
- Creates new collections with proper vectorConfig including "default" named vector
- Migrates data using cursor-based pagination (efficient for large datasets)
- Uses batch operations for fast inserts
- Preserves all object properties and vectors
- (Update) Fixed 10k object limit issue
- (Update) Added timestamps to logs
"""
 
import weaviate
from weaviate.classes.config import Configure, VectorDistances
import sys
import time
from typing import List, Dict, Any
import requests
from datetime import datetime
 
# Configuration
WEAVIATE_HOST = "<your weaviate container ip addr>"
WEAVIATE_PORT = 8080
WEAVIATE_GRPC_PORT = 50051
WEAVIATE_API_KEY = "WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih"
BATCH_SIZE = 10000
 
 
def log_msg(message: str):
    """Print message with current timestamp"""
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"[{timestamp}] {message}")
 
 
def identify_old_collections(client: weaviate.WeaviateClient) -> List[str]:
    """Identify collections that need migration (those without vectorConfig)"""
    collections_to_migrate = []
 
    all_collections = client.collections.list_all()
    log_msg(f"Found {len(all_collections)} total collections")
 
    for collection_name in all_collections.keys():
        # Only check Vector_index collections (Dify knowledge bases)
        if not collection_name.startswith("Vector_index_"):
            continue
 
        collection = client.collections.get(collection_name)
        config = collection.config.get()
 
        # Check if this collection has the old schema
        if config.vector_config is None:
            collections_to_migrate.append(collection_name)
            log_msg(f"  - {collection_name}: OLD SCHEMA (needs migration)")
        else:
            # log_msg(f"  - {collection_name}: NEW SCHEMA (skip)") # Optional: Comment out to reduce noise
            pass
 
    return collections_to_migrate
 
 
def get_collection_schema(client: weaviate.WeaviateClient, collection_name: str) -> Dict[str, Any]:
    """Get the full schema of a collection via REST API"""
 
    response = requests.get(
        f"http://{WEAVIATE_HOST}:{WEAVIATE_PORT}/v1/schema/{collection_name}",
        headers={"Authorization": f"Bearer {WEAVIATE_API_KEY}"}
    )
 
    if response.status_code == 200:
        return response.json()
    else:
        raise Exception(f"Failed to get schema: {response.text}")
 
 
def create_new_collection(client: weaviate.WeaviateClient, old_name: str, schema: Dict[str, Any]) -> str:
    """Create a new collection with updated schema using REST API"""
 
    # Generate new collection name
    new_name = f"{old_name}_migrated"
 
    log_msg(f"Creating new collection: {new_name}")
 
    # Build new schema with proper vectorConfig
    # Note: When using vectorConfig (named vectors), we don't set class-level vectorizer
    new_schema = {
        "class": new_name,
        # This is the key: define vectorConfig with "default" named vector
        # Do NOT set class-level vectorizer when using vectorConfig
        "vectorConfig": {
            "default": {
                "vectorizer": {
                    "none": {}
                },
                "vectorIndexType": "hnsw",
                "vectorIndexConfig": {
                    "distance": "cosine",
                    "ef": -1,
                    "efConstruction": 128,
                    "maxConnections": 32
                }
            }
        },
        "properties": []
    }
 
    # Copy properties from old schema
    if "properties" in schema:
        new_schema["properties"] = schema["properties"]
 
    # Create collection via REST API
    response = requests.post(
        f"http://{WEAVIATE_HOST}:{WEAVIATE_PORT}/v1/schema",
        json=new_schema,
        headers={"Authorization": f"Bearer {WEAVIATE_API_KEY}"}
    )
 
    if response.status_code not in [200, 201]:
        raise Exception(f"Failed to create collection: {response.text}")
 
    log_msg(f"  Created new collection: {new_name}")
    return new_name
 
 
def migrate_collection_data(
    client: weaviate.WeaviateClient,
    old_collection_name: str,
    new_collection_name: str
) -> int:
    """Migrate data from old collection to new collection using cursor-based pagination"""
 
    old_collection = client.collections.get(old_collection_name)
    new_collection = client.collections.get(new_collection_name)
 
    total_migrated = 0
    cursor = None
 
    log_msg(f"Migrating data from {old_collection_name} to {new_collection_name}")
 
    while True:
        # Fetch batch of objects using cursor-based pagination
        if cursor is None:
            # First batch
            response = old_collection.query.fetch_objects(
                limit=BATCH_SIZE,
                include_vector=True
            )
        else:
            # Subsequent batches using cursor
            response = old_collection.query.fetch_objects(
                limit=BATCH_SIZE,
                include_vector=True,
                after=cursor
            )
 
        objects = response.objects
 
        if not objects:
            break
 
        # Use batch insert for efficiency
        with new_collection.batch.dynamic() as batch:
            for obj in objects:
                # Prepare properties
                properties = obj.properties
 
                # Add object with vector
                # Handle potential dict vector (named) or list vector (legacy)
                vector_data = obj.vector["default"] if isinstance(obj.vector, dict) else obj.vector
 
                batch.add_object(
                    properties=properties,
                    vector=vector_data,
                    uuid=obj.uuid
                )
 
        total_migrated += len(objects)
        log_msg(f"  Migrated {total_migrated} objects...")
 
        # Update cursor for next iteration
        if len(objects) < BATCH_SIZE:
            # Last batch
            break
        else:
            # Get the last object's UUID for cursor
            cursor = objects[-1].uuid
 
    log_msg(f"  Total migrated: {total_migrated} objects")
    return total_migrated
 
 
def verify_migration(
    client: weaviate.WeaviateClient,
    old_collection_name: str,
    new_collection_name: str
):
    """Verify that the migration was successful"""
 
    old_collection = client.collections.get(old_collection_name)
    new_collection = client.collections.get(new_collection_name)
 
    # Get aggregation for accurate counts
    old_agg = old_collection.aggregate.over_all(total_count=True)
    new_agg = new_collection.aggregate.over_all(total_count=True)
 
    old_count = old_agg.total_count
    new_count = new_agg.total_count
 
    log_msg(f"Verification:")
    log_msg(f"  Old collection ({old_collection_name}): {old_count} objects")
    log_msg(f"  New collection ({new_collection_name}): {new_count} objects")
 
    if old_count == new_count:
        log_msg(f"  Status: SUCCESS - Counts match!")
        return True
    else:
        log_msg(f"  Status: WARNING - Counts don't match!")
        return False
 
 
def replace_old_collection(
    client: weaviate.WeaviateClient,
    old_collection_name: str,
    new_collection_name: str
):
    """Replace old collection with migrated one by recreating with original name"""
 
    log_msg(f"Replacing old collection with migrated data...")
 
    # Step 1: Get reference to migrated collection
    migrated = client.collections.get(new_collection_name)
    migrated_count = migrated.aggregate.over_all(total_count=True).total_count
    log_msg(f"  Step 1: Migrated collection has {migrated_count} objects")
 
    # Step 2: Delete old collection
    log_msg(f"  Step 2: Deleting old collection...")
    response = requests.delete(
        f"http://{WEAVIATE_HOST}:{WEAVIATE_PORT}/v1/schema/{old_collection_name}",
        headers={"Authorization": f"Bearer {WEAVIATE_API_KEY}"}
    )
    if response.status_code != 200:
        log_msg(f"    Warning: Could not delete old collection: {response.text}")
    else:
        log_msg(f"    Deleted")
 
    # Step 3: Get schema from migrated collection
    log_msg(f"  Step 3: Getting schema from migrated collection...")
    schema_response = requests.get(
        f"http://{WEAVIATE_HOST}:{WEAVIATE_PORT}/v1/schema/{new_collection_name}",
        headers={"Authorization": f"Bearer {WEAVIATE_API_KEY}"}
    )
    schema = schema_response.json()
    schema["class"] = old_collection_name
 
    # Step 4: Create collection with original name and new schema
    log_msg(f"  Step 4: Creating collection with original name...")
    create_response = requests.post(
        f"http://{WEAVIATE_HOST}:{WEAVIATE_PORT}/v1/schema",
        json=schema,
        headers={"Authorization": f"Bearer {WEAVIATE_API_KEY}"}
    )
    if create_response.status_code not in [200, 201]:
        raise Exception(f"Failed to create collection: {create_response.text}")
    log_msg(f"    Created")
 
    # Step 5: Copy data to collection with original name (Corrected for large datasets)
    log_msg(f"  Step 5: Copying data to original collection name...")
    new_collection = client.collections.get(old_collection_name)
 
    cursor = None
    total_copied = 0
 
    while True:
        # Fetch batch of objects using cursor-based pagination
        if cursor is None:
            response = migrated.query.fetch_objects(
                limit=BATCH_SIZE,
                include_vector=True
            )
        else:
            response = migrated.query.fetch_objects(
                limit=BATCH_SIZE,
                include_vector=True,
                after=cursor
            )
 
        objects = response.objects
 
        if not objects:
            break
 
        # Batch insert
        with new_collection.batch.dynamic() as batch:
            for obj in objects:
                batch.add_object(
                    properties=obj.properties,
                    vector=obj.vector,
                    uuid=obj.uuid
                )
 
        total_copied += len(objects)
        log_msg(f"    Copied {total_copied} objects...")
 
        # Update cursor or break
        if len(objects) < BATCH_SIZE:
            break
        else:
            cursor = objects[-1].uuid
 
    count = new_collection.aggregate.over_all(total_count=True).total_count
    log_msg(f"    Copied {count} objects")
 
    # Step 6: Delete the temporary migrated collection
    log_msg(f"  Step 6: Cleaning up temporary migrated collection...")
    response = requests.delete(
        f"http://{WEAVIATE_HOST}:{WEAVIATE_PORT}/v1/schema/{new_collection_name}",
        headers={"Authorization": f"Bearer {WEAVIATE_API_KEY}"}
    )
    if response.status_code == 200:
        log_msg(f"    Cleaned up")
 
    log_msg(f"SUCCESS! {old_collection_name} now has the new schema with {count} objects")
    return True
 
 
def migrate_all_collections():
    """Main migration function"""
 
    print("=" * 80)
    print("Weaviate Collection Migration Script (Fixed for large datasets + Timestamp)")
    print("Migrating from Weaviate 1.19.0 schema to 1.27.0+ schema")
    print("=" * 80)
    print()
 
    client = weaviate.connect_to_local(
        host=WEAVIATE_HOST,
        port=WEAVIATE_PORT,
        grpc_port=WEAVIATE_GRPC_PORT,
        auth_credentials=weaviate.auth.AuthApiKey(WEAVIATE_API_KEY)
    )
 
    try:
        # Step 1: Identify collections that need migration
        log_msg("Step 1: Identifying collections that need migration...")
        collections_to_migrate = identify_old_collections(client)
 
        if not collections_to_migrate:
            log_msg("No collections need migration. All collections are up to date!")
            return
 
        log_msg(f"Found {len(collections_to_migrate)} collections to migrate:")
        for col in collections_to_migrate:
            print(f"  - {col}")
 
        # Confirm before proceeding
        print("\nThis script will:")
        print("1. Create new collections with updated schema")
        print("2. Copy all data using efficient batch operations")
        print("3. Verify the migration")
        print("4. Automatically rename collections to activate the new ones")
        print()
 
        # Step 2: Migrate each collection
        for collection_name in collections_to_migrate:
            print("\n" + "=" * 80)
            log_msg(f"Migrating: {collection_name}")
            print("=" * 80)
 
            try:
                # Get old schema
                schema = get_collection_schema(client, collection_name)
 
                # Create new collection
                new_collection_name = create_new_collection(client, collection_name, schema)
 
                # Migrate data
                migrated_count = migrate_collection_data(client, collection_name, new_collection_name)
 
                # Verify migration
                success = verify_migration(client, collection_name, new_collection_name)
 
                if success and migrated_count > 0:
                    log_msg(f"Migration successful for {collection_name}!")
                    log_msg(f"New collection: {new_collection_name}")
 
                    # Automatically replace old collection with migrated one
                    try:
                        replace_old_collection(client, collection_name, new_collection_name)
                    except Exception as e:
                        log_msg(f"Warning: Could not automatically replace collection: {e}")
                        print(f"\nTo activate manually:")
                        print(f"1. Delete the old collection: {collection_name}")
                        print(f"2. Rename {new_collection_name} to {collection_name}")
 
            except Exception as e:
                log_msg(f"Error migrating {collection_name}: {e}")
                log_msg(f"Skipping this collection and continuing...")
                import traceback
                traceback.print_exc()
                continue
 
        print("\n" + "=" * 80)
        log_msg("Migration Complete!")
        print("=" * 80)
        print("\nSummary:")
        print(f"  Collections migrated: {len(collections_to_migrate)}")
 
    finally:
        client.close()
 
 
if __name__ == "__main__":
    try:
        migrate_all_collections()
    except KeyboardInterrupt:
        print("\n\nMigration interrupted by user.")
        sys.exit(1)
    except Exception as e:
        print(f"\n\nFatal error: {e}")
        import traceback
        traceback.print_exc()
        sys.exit(1)

@jmjoy
Thanks for the report! Coincidentally, I tried migrating the environment I manage today and got the same warning.
It seems the gRPC max message size can only be set via environment variables from Weaviate 1.27.1. Since Dify bundles 1.27.0, that option doesn’t work.
In the end, like @tomy-kyu, I feel that cursor-based pagination might be necessary. I’ll experiment a bit and try updating the script.

@tomy-kyu
Thanks for sharing your updated script! I’ll give it a try.

@jmjoy @tomy-kyu
I've updated the script. Since a batch size of 10,000 might exceed GRPC_MAX_MESSAGE_SIZE, I've set it to 1,000 just to be safe.

Handling replace_old_collection based on pagination feels a bit risky to me, because it requires deleting the old class first. If something goes wrong during the paginated migration process, you’d need ways to recover or retry, which is a real hassle.

The root of all these problems is that Weaviate somehow doesn’t support what should be the most basic database operation: renaming.

@jmjoy

I apologize for presenting an unusual processing example.
Ah, now I understand why we were seeing the following warning message:

Warning: Could not automatically replace collection: Query call with protocol GRPC search failed with the message "grpc: trying to send message larger than max (46033918 vs. 10485760)."

To manually resolve this issue:
1. Delete the old collection: Vector_index_015e0da7_4e44_4715_9127_8142873eea25_Node
2. Rename Vector_index_015e0da7_4e44_4715_9127_8142873eea25_Node to Vector_index_015e0da7_4e44_4715_9127_8142873eea25_Node_migrated

So the solution is to manually rename the collection using OS commands or similar methods?

Based on this, it seems better not to modify GRPC_MAX_MESSAGE_SIZE. Instead, we should intentionally set GRPC_MAX_MESSAGE_SIZE above its maximum value to actively trigger the above warning.

The reason is that if we pass the expanded GRPC_MAX_MESSAGE_SIZE to the next processing stage, the replace_old_collection operation will still apply the limit=10000 constraint, causing any data exceeding this limit to be truncated—potentially leading to the accidental deletion of temporary collections as well.

I believe this important consideration should be noted explicitly in the documentation.

@jmjoy @tomy-kyu
To summarize, the script firstly created by the Dify Team migrates each collection in the following two steps (1️⃣ and 2️⃣):

Vector_index_* (Original collection with old schema)
  ↓
  ↓  1️⃣ Migrate all objects in batches of 100 using pagination.
  ↓
Vector_index_*_migrated (New collection with new schema)
  ↓
  ↓  2️⃣ Migrate 10,000 objects all at once
  ↓    🚨 If there are more than 10,000 objects, the extra objects will be truncated.
  ↓    🚨 The GRPC_MAX_MESSAGE_SIZE could be exceeded causing a warning will be triggered and the migration may be interrupted.
  ↓
Vector_index_* (Re-created new collection with new schema)

Based on your comments, I have already modified the script in this gist as follows:

Vector_index_* (Original collection with old schema)
  ↓
  ↓  1️⃣ Migrate all objects in batches of 1,000 using pagination.
  ↓    ✅ Update batch size from 100 to 1,000 for faster migration
  ↓
Vector_index_*_migrated (New collection with new schema)
  ↓
  ↓  2️⃣ Migrate all objects in batches of 1,000 using pagination.
  ↓    ✅ Use pagenation to support large collection
  ↓
Vector_index_* (Re-created new collection with new schema)

Even when using pagination, if the batch size is set to 10,000, there's a possibility of exceeding GRPC_MAX_MESSAGE_SIZE, so in my script, I set it to 1,000.

@jmjoy
Agree with you, but the function replace_old_collection is used for step 2️⃣.
It first deletes the collection, which targets the old collection with the old schema. Even if an error occurs mid-pagination, as long as all the data has been persistently stored in Vector_index*migrated (by step 1️⃣), it should be technically possible to recover by intervening manually (though, to be fair, the script is not automated to handle that).
Also, since the old directory still remains in volmes, you can start the migration over from the beginning if needed.

Ah, so if the write-back process fails and stops, the subsequent migration collection deletion operation won't occur. In that case, you should:

To manually resolve this issue:
1. Delete the old collection: Vector_index_*_Node
2. Rename Vector_index_*_Node to Vector_index_*_Node_migrated

This appears to be the correct approach.

My primary concern was that when running the initial script in an environment where gRPC-related configuration values had been modified, the replace_old_collection operation would truncate most of the larger index data and completely remove the primary collection data, completing the process without error. This would also result in the loss of migrated data, leaving only the option to restore from backup.

While 10,000 objects might seem insignificant in Japanese terms, with only about 416,000 characters of capacity, simply including a few documents would easily exceed this limit. In my environment, this was unacceptable, so I incorporated a similar migration workflow into the replace operation.

In particular, knowledge data often contains numerous documents and may naturally have large index sizes - this example was provided as reference material for such cases.

@kurokobo
Thank you for your hard work on these modifications.
This procedure contains more detailed steps than the original plan, so when actually implementing the work, I plan to refer to this guide as a reference. If this proves to be of any help to you, I would be truly grateful.

One question I have for confirmation: In Step 3, you're starting the DIFY-related containers updated to version 1.9.2. Is this done so that the Worker-side can access the updated Weaviate client when performing operations in Step 4?

I also wondered whether starting the API also triggers schema updates in PostgreSQL, so perhaps you launched all services at once to ensure synchronization of update timing. I'd appreciate it if you could confirm this for me.

@tomy-kyu
Thank you for your positive comment.

One question I have for confirmation: In Step 3, you're starting the DIFY-related containers updated to version 1.9.2. Is this done so that the Worker-side can access the updated Weaviate client when performing operations in Step 4?

I also wondered whether starting the API also triggers schema updates in PostgreSQL, so perhaps you launched all services at once to ensure synchronization of update timing. I'd appreciate it if you could confirm this for me.

Your understanding is almost correct.

Technically, PostgreSQL migration is completely independent from Weaviate migration, and there is no dependency between them.
Therefore, when considering the migration steps for Weaviate, there is no need to take the PostgreSQL migration into account.

However, as you mentioned, Weaviate migration requires a new client, which is deployed in the Worker, API, and Beat containers. Of course, it's also possible to set up the Weaviate client on a user's terminal (as the official documentation suggests), but that's not necessarily a recommended approach for everyone.

Given this background, I thought that updating the API, Worker, Beat, and Weaviate all at once would be the most reasonable, simple, and least problematic migration process.

Hope this helps :)

@kurokobo
Unfortunately, the migration has not been successful.
It seems some trial and error may be required.

Due to the gRPC migration, it appears that after switching Weaviate to gRPC, the index_node_id returned from the vector database is now being treated as a UUID object rather than a string.
This has caused SQLAlchemy to generate UUID-type parameters, and PostgreSQL is failing to perform the type conversion.
Consequently, the following error was being output:

Run failed: (psycopg2.errors.UndefinedFunction) operator does not exist: character varying = uuid
LINE 1: ...T_TIMESTAMP WHERE document_segments.index_node_id = 'c6f0cc1...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.

[SQL: UPDATE document_segments SET hit_count=(document_segments.hit_count + %(hit_count_1)s), updated_at=CURRENT_TIMESTAMP WHERE document_segments.index_node_id = %(index_node_id_1)s::UUID AND document_segments.dataset_id = %(dataset_id_1)s::UUID]
[parameters: {'hit_count_1': 1, 'index_node_id_1': UUID('c6f0cc12-0522-41b5-b22f-55e8dcbdc79a'), 'dataset_id_1': 'c04750cd-58e6-4e24-85d5-852dd3c4d321'}]
(Background on this error at: https://sqlalche.me/e/20/f405)

Therefore, during the migration process, explicit modifications should be made to return UUID values as strings instead. While I plan to try implementing these corrections, I wanted to note that this approach appears unworkable in its current state, so I'm leaving this comment for reference.

@tomy-kyu
Could you please provide the exact steps to reproduce your issue?
After the migration, what operations are you performing, and where is the error being logged?

In any case, it would be best to clarify whether this is a problem with this guide itself or if it could also happen when following the official guide. If necessary, we shouldn't try to resolve everything here, but instead open an issue at official repo.

@kurokobo
Correct. We're currently running the correction script, and if it executes successfully, we'll include this information as reference. The sequence of events was as follows:

• Used this guide to migrate from 1.8.1 to 1.9.2
• After execution, tested queries on the knowledge base screen and confirmed they returned zero results
• Executed the knowledge search node in the workflow app
• The problematic error occurred

@tomy-kyu
Thanks, I tried to reproduce your issue, but I couldn't.

1. Deploy a whole new Dify 1.8.1 instance, install the OpenAI plugin, and configure the OpenAI API key.
2. Create new Knowledge with the High Quality method using the OpenAI embedding model.
3. Ensure that the Retrieval Testing works.
4. Create new workflow app including Knowledge Retrieval node with the Knowledge created above.
5. Ensure that the new app can retrieve the Knowledge, and retrieval count increased.
6. Follow my guide to upgrade Dify to 1.9.2 and Weaviate to 1.27.0.

Now I can confirm that both the Retrieval Testing and the workflow app works, and retrieval count increased as expected.
Adding docs to the existing knowledge also works.

@kurokobo

Thank you for your assistance with the verification process.
I've now understood that under normal circumstances, your system should function correctly.

My environment does indeed have some differences:

• Weaviate was originally configured with the latest version 1.28 during initial setup.
• We customized several processing steps to use gse as the tokenizer.
• Our knowledge configuration always utilized parent-child chunks.

What occurred in my environment may be attributable to these factors.
Given the practical challenges of completely restoring and reimplementing all these changes at this time, I'm considering continuing to operate the tenant on version 1.10.1.fix1 for the moment.

Fortunately, we have two tenants (production and development), and the target is the development tenant. I expect the development tenant will continue to serve as a testing environment for versions 1.10 and above.

The production tenant remains on version 1.8.1. I'll freeze this version for ongoing operation for the time being, then consider setting up a new production tenant as needed and performing an environment migration.

I apologize for not being able to provide more useful information.
Best wishes for your future endeavors.

run: uv run /app/api/storage/migrate_weaviate_collections_ce.py
error:
[
](error: failed to create directory /home/dify/.cache/uv: Permission denied (os error 13))
fix:
docker compose exec -it -u root worker /bin/bash
uv run /app/api/storage/migrate_weaviate_collections_ce.py

@suntao2015005848
Good catch, thanks!
It seems that /home/dify no longer exists in the container starting from version 1.11.0, which caused the cache directory for uv to fail to be created.

As you suggested, using -u root would work, but an even simpler solution is just to add --no-cache to the uv command. I've updated the guide accordingly.

Thanks again!