Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration of the scheduler and orchestrator packages #65

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

augustocristian
Copy link
Contributor

This PR is related and closes #58 contains the following changes:

  • The scheduler and orchestrator packages, with the necessary code to generate the TJobs and Execution plans
  • The necessary objects in the model folder
  • The unitary test cases of both components
  • The integration test of the whole orchestration generator.

@augustocristian augustocristian linked an issue Dec 13, 2024 that may be closed by this pull request
@augustocristian augustocristian force-pushed the ft_migrateschedulerorchestrator branch from 1aba782 to 937bb4b Compare December 13, 2024 18:07
@augustocristian
Copy link
Contributor Author

augustocristian commented Dec 13, 2024

As you requested @javiertuya #59 (review), I've migrated the module that generates the Jenkinsfiles (nothing related to the UsageProfile or the Cost estimation).
I've refactored a lot and the current version is far away to the ones that are in GitLab, I've removed a lot of duplications and dummy methods not used. Looks like that there are a lot of new files, but most of them are base files to generate the jenkinsfile and desired outputs for the test cases.

- All the scripts with comments and added some description also to the .env files
- Removed unnecesary comments in pom.xml
- Removed all the references to "tool" or "package"
- All the scripts with comments and added some description also to the .env files
- Removed unnecesary comments in pom.xml
- Removed all the references to "tool" or "package"
- Adding the I to the interface name
- Removed the enum and created a method that generates dinamically an alphabetical identifier.
- Added Jenkinsfile generator to the main
- Aligned the prior methods
- Removed capital
- Added Jenkinsfile generator to the main
- Aligned the prior methods
- Removed capital
Copy link
Contributor

@javiertuya javiertuya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review done, but problems generating the jenkinsfile according to the documentation.

README.md Outdated Show resolved Hide resolved
@@ -223,25 +223,11 @@ Once created the different properties and configuration files, the single module
```

### Executing the Orchestration generator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to execute the main method from eclipse to generate fullteaching (fullteaching repo is a sibling of retorch). This is what I see:

  1. Logs are not shown:
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
  1. Executing with parameters: "com.fullteaching.e2e.no_elastest.common","FullTeaching","../../retorch-st-fullteaching"
    does not find the resource file: `Exception in thread "main" java.nio.file.NoSuchFileException: retorchfiles\configurations\FullTeachingSystemResources.json

Copy link
Contributor Author

@augustocristian augustocristian Jan 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javiertuya
I was experiencing some problems with the IDE during the past week, I've managed more or less to solve it (working with the LAPTOP and DESKTOP in parallel). Some remarks (that I don't understand):

  1. FullTeaching was with the old orchestration generation tool, which is why you don't find the Resources file or the configuration file. Now, the ft_AlignOrchGeneratorVersion is fully updated and working.
  2. I've created a class with a simple call to the main in the FullTeaching project, importing the snapshot orchestration using the dependency:
        <dependency>
            <groupId>io.github.giis-uniovi</groupId>
            <artifactId>retorch-orchestration</artifactId>
            <version>1.1.1-ft_migrateschedulerorchestrator-SNAPSHOT</version>
        </dependency>

The class looks like this:

package com.fullteaching.e2e.no_elastest.functional.test;

import giis.retorch.orchestration.classifier.EmptyInputException;
import giis.retorch.orchestration.generator.OrchestationGeneratorMainClass;
import giis.retorch.orchestration.orchestrator.NoFinalActivitiesException;
import giis.retorch.orchestration.scheduler.NoTGroupsInTheSchedulerException;
import giis.retorch.orchestration.scheduler.NotValidSystemException;

import java.io.IOException;
import java.net.URISyntaxException;

public class RetorchMain {
    public static void main(String[] args) throws NoFinalActivitiesException, NoTGroupsInTheSchedulerException, EmptyInputException, IOException, URISyntaxException, NotValidSystemException, ClassNotFoundException {
        String [] argum={"com.fullteaching.e2e.no_elastest.functional.test", "FullTeaching", "./"};
        OrchestationGeneratorMainClass.main(argum);

    }
}

I've also added some test cases to check that the scripts generated are correct and solved some problems with the current implementation of the execution plan generation.
If I add the path com.fullteaching.e2e.no_elastest.functional, it fails because It finds java classes without annotations, Should I change the actual implementation to allow this behavior?
If I understood well, you have the following directories:

.
├── retorch/
│   ├── annotations
│   └── orchestration/
│       └── -YourMainClass
└── retorch-st-fullteaching

And you're trying to call RETORCHOrchestrationGeneratorMain.main() from YourMainClass in orchestration folder. I am pretty sure that it's not working because is trying to find the Resources.json and retorchCI.files (as well as the docker-compose.yml) in the orchestration folder (not in retorch-st-fullteaching). I can change the orchestration generator to provide the basepath, and allow the use case that you're trying.

- Adding the validation to the tjob.env files
- Adding some support methods to validate the files
- Changed the delimitation character (from [IMG:] to ;
- Some minor typos in the ExecutionPlan.java
- Adjusting tjob and orchestration outputs
- Fixing the full qualified class name
- Reviewed that all paths to the files matchs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Start migration of retorch-orchestration tool
2 participants