Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] CreateIngestPipelineStep Improvements #104

Closed
1 task
Tracked by #475 ...
joshpalis opened this issue Oct 18, 2023 · 1 comment
Closed
1 task
Tracked by #475 ...

[FEATURE] CreateIngestPipelineStep Improvements #104

joshpalis opened this issue Oct 18, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request v2.13.0 Issues targeting release v2.13.0

Comments

@joshpalis
Copy link
Member

joshpalis commented Oct 18, 2023

Is your feature request related to a problem?

In an effort to address any issues/gaps with the current step implementations, this issue is to track necessary modifications for the CreateIngestPipelineStep.

  • The use case template workflow parser is responsible for populating the WorkflowData that each WorkflowStep will read information from to execute their function. Modify how the CreateIngestPipelineStep currently reads data from the WorkflowData to ensure that the expected information is read correctly. The following example WorkflowData is what is generated from the following use case template section.

Use Case Template Create Ingest Pipeline Section :

{
    "id": "create_ingest_pipeline",
    "type": "create_ingest_pipeline",
    "inputs": {
      "id": "my-ingest-pipeline",
      "description": "some description",
      "processors": [
        {
          "type": "text_embedding",
          "params": {
            "model_id": "my-existing-model-id",
            "input_field_name": "text_passage",
            "output_field_name": "text_embedding"
          }
        }
      ]
    }
  }

WorkflowData :

{
    description=some description,
    id=my-ingest-pipeline, 
    processors=[Lorg.opensearch.flowframework.model.PipelineProcessor;
}

Note : The PipelineProcessor (link) contains the type and a list of required parameters for this type. In the case of the text-embedding processor, the required parameters are the model_id, 'input_field_name', 'output_field_name'.

@amitgalitz
Copy link
Member

Done with #558

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request v2.13.0 Issues targeting release v2.13.0
Projects
None yet
Development

No branches or pull requests

3 participants