Having problem using Artifact to store data in Minio, help needed

[LehmBook]

Context:
Im a beginner in linux and kubeflow, and im trying to use Artifact to pass data between components

env:
kubenetes 1.30.3
kubeflow :1.9.1
base image:python 3.8, Tensorflow:2.15.0, Kfp 2.8.0

procedure:
having the py file as follow, get yaml, upload and run

code:

from kfp import dsl
from kfp.dsl import Output, Dataset

@dsl.component( base_image="harbor-local.com:5012/pipeline/mnist_load:v3"
)
def generate_data(output_data: Output[Dataset]):
import numpy as np
np.savez_compressed(output_data.path, data=np.random.rand(100, 10))

@dsl.pipeline
def test_pipeline():
generate_data_task = generate_data()

if name == "main":
from kfp import compiler
compiler.Compiler().compile(test_pipeline, "simple_test.yaml")`

what happens:
the logs as follow:

> `[KFP Executor 2024-11-20 08:20:52,530 INFO]: Looking for component `generate_data` in --component_module_path `/tmp/tmp.3FKA82NfyG/ephemeral_component.py`
> [KFP Executor 2024-11-20 08:20:52,530 INFO]: Loading KFP component "generate_data" from /tmp/tmp.3FKA82NfyG/ephemeral_component.py (directory "/tmp/tmp.3FKA82NfyG" and module name "ephemeral_component")
> [KFP Executor 2024-11-20 08:20:52,531 INFO]: Got executor_input:
> {
>     "inputs": {},
>     "outputs": {
>         "artifacts": {
>             "output_data": {
>                 "artifacts": [
>                     {
>                         "type": {
>                             "schemaTitle": "system.Dataset",
>                             "schemaVersion": "0.0.1"
>                         },
>                         "uri": "minio://mlpipeline/v2/artifacts/test-pipeline/1b5960d0-e8c7-4206-82b2-4dff60f5d22f/generate-data/output_data"
>                     }
>                 ]
>             }
>         },
>         "outputFile": "/tmp/kfp_outputs/output_metadata.json"
>     }
> }
> [KFP Executor 2024-11-20 08:20:52,640 INFO]: Wrote executor output file to /tmp/kfp_outputs/output_metadata.json.
> /usr/local/lib/python3.8/runpy.py:111: FutureWarning: KFP will drop support for Python 3.8 on Oct 1, 2024. To use new versions of the KFP SDK after that date, you will need to upgrade to Python >= 3.9. See 
> https://devguide.python.org/versions/
>  for more details.
>   __import__(pkg_name)
> I1120 08:20:52.667567      25 launcher_v2.go:704] ExecutorOutput: {
>   "artifacts": {
>     "output_data": {
>       "artifacts": [
>         {
>           "name": "",
>           "uri": "minio://mlpipeline/v2/artifacts/test-pipeline/1b5960d0-e8c7-4206-82b2-4dff60f5d22f/generate-data/output_data",
>           "metadata": {}
>         }
>       ]
>     }
>   }
> }
> W1120 08:20:52.667893      25 launcher_v2.go:445] Local filepath "/minio/mlpipeline/v2/artifacts/test-pipeline/1b5960d0-e8c7-4206-82b2-4dff60f5d22f/generate-data/output_data" does not exist
> I1120 08:20:52.988517      25 launcher_v2.go:150] publish success.
> time="2024-11-20T08:20:53.862Z" level=info msg="sub-process exited" argo=true error="<nil>"`

And i logged in minio and did not find the dir named /v2/artifacts/.... so no file was stored in the minio. and I checked the secret "mlpipeline-minio-artifact" can be found in my working namespace, I would like to know how to fix it or whats the correct way to use Artifacts to pass output data as the input of the next component, many thanks

[LehmBook]

after some time debugging, I fould out its a stupid question:

by writing np.savez_compressed(output_data.path, data=np.random.rand(100, 10))
the file actually saved as
/minio/mlpipeline/v2/artifacts/test-pipeline/1b5960d0-e8c7-4206-82b2-4dff60f5d22f/generate-data/output_data.npz

but when artifact uploading file to minio here, it only support file with out extension which is
/minio/mlpipeline/v2/artifacts/test-pipeline/1b5960d0-e8c7-4206-82b2-4dff60f5d22f/generate-data/output_data

How to fix it:

    with open(output_data.path, 'wb') as f:
        np.savez_compressed(output_data.path, data=np.random.rand(100, 10))

this way the file saved in correct format