Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Support Embeddings in mltransform (#29564)
* Make base.py framework agnostic and add helper transforms * Add tests for base.py * Add sentence-transformers * Add tensorflow hub * Add vertex_ai * Make TFTProcessHandler a PTransform * raise RuntimeError in ArtifactsFetcher when it is used for embeddings * Add JsonPickle to requirements * Add tox tests * Mock frameworks in pydocs Fix tox.ini Fix pydoc Fix indent in pydoc * Add Row type check * Remove requires_chaining * change name of PTransformProvider to MLTransformProvider * remove batch_len in utility fun * Change type annotation and redundant comments * Remove get_transforms method * remove requires_chaining from tft * add tests to sentence-transformers * Pass inference_args to RunInference * Add TODO GH issue * refactor variables in vertex_ai embeddings * remove try/catch and throw error if options is empty for GCS artifact location * Refactor NotImplementedError message * remove tensorflow hub from this PR * Add _validate_transform method * add more tests * fix test * Fix test * Add more tests in sentence-transformer * use np.max instead of max * round to 2 decimals * Remove gradle command action * Refactor throwing dataflow client exception * skip the test if gcp is not installed * remove toxTests for hub * remove toxTests for hub * Fix values in assert for sentence_transformer_test * rename sentence_transformers to huggingface * fix pydocs * Change the model name for tests since it is getting different results on different machines * Fix pydoc in vertexai * add suffix to artifact_location * Revert "add suffix to artifact_location" This reverts commit cfb1883. * add no_xdist * Try fixing pydoc for vertexai * change tox.ini to use pytest directly * raise FileExistError if Attribute file is already present * modify build.gradle to match tox task names * Add note to CHANGES.md * change gcs bucket to gs://temp-storage-for-perf-tests * Add TODO GH links * Update CHANGES.md Co-authored-by: Danny McCormick <[email protected]> --------- Co-authored-by: Danny McCormick <[email protected]>
- Loading branch information