[WIP] Add e2e test for tune api with LLM hyperparameter optimization

[helenxie-bit]

What this PR does / why we need it:
This PR adds E2E tests for the tune API in the following scenarios:

• Creating Katib Experiment with custom objective using distributed training with PyTorchJob.
• Creating Katib Experiment with external models and datasets using non-distributed training.
• Creating Katib Experiment with external models and datasets using distributed training with PyTorchJob.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

• For a detailed explanation of this API, please refer to the proposal: [GSoC] Create LLM Hyperparameters Optimization API Proposal #2333

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[helenxie-bit]

/area gsoc

[helenxie-bit]

Ref: #2339