[Bug]: faile to export all data

[minglong-huang]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:milvusdb/milvus:v2.4.0 
- Deployment mode(standalone or cluster):standalone 
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus  v2.5.0
- OS(Ubuntu or CentOS): Ubuntu 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Firstly, I installed milvus and ran it for a month using the following method:
wget https://github.com/milvus-io/milvus/releases/download/v2.4.0/milvus-standalone-docker-compose.yml -O docker-compose.yml
sudo docker-compose up -d
Then in recent days,I try export data into graphrag format,But I found it faile to export all data
Here is my code

from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection,utility
import pandas as pd
from xinference.client import Client
from tqdm import tqdm

client = Client("http://192.0.0.181:9997")
list_models_run = client.list_models()
model_uid = list_models_run['bge-m3']['id']
embedding_client = client.get_model(model_uid)

connections.connect(host='0.0.0.0', port="19530")
collection = Collection(name='temp')
entities = collection.load()

pk = collection.schema.primary_field.name
vector_field_name = collection.indexes[0].field_name

batch_size = 10
output_fields = ["id", "text", "file_name", "text_embedding"]
query_iter = collection.query_iterator(
    batch_size=batch_size,
    output_fields=output_fields,
    partition_names=['_default']
)

new_doc_file = []
pabr = tqdm(total=collection.num_entities)
while True:
    docs = query_iter.next()
    if len(docs) == 0:
        # close the iterator
        query_iter.close()
        break
    for doc in docs:
        new_doc = {}
        for k, v in doc.items():
            if k =='file_name':
                new_doc['document_ids'] = [v]
            elif k == 'text':
                new_doc[k] = v
                n_tokens = embedding_client.create_embedding(v)['usage']['total_tokens']
                new_doc['n_tokens'] = n_tokens
            else:
                new_doc[k] = v
        new_doc['entity_ids'] = []
        new_doc['relationship_ids'] = []

        new_doc_file.append(new_doc)
    pabr.update(batch_size)


df = pd.DataFrame(new_doc_file)
df.to_parquet('/home/nlp/temp/test_1.parquet', engine='pyarrow')

num_entities = 10104 But really export 9753

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[yanliang567]

please retry with Milvus 2.5.0, which had an improvement for query_iterator()

/assign @minglong-huang
/unassign