Please support File References

[philCryoport]

Reason/Context

Please try answering few of those questions

• Why we need this improvement?
  • AsyncAPI files can get very long very quickly -- which makes readability difficult
  • It would be great to be able to extract out a collection of common schemas and then use it in multiple files

Description

Please try answering few of those questions

• What changes have to be introduced?
  • Be able to load multiple files -- preferably an entire subdirectory tree -- into the editor
• Will this be a breaking change?
  • Unknown
• How could it be implemented/designed?
  • Unknown

[github-actions]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[magicmatatjahu]

@philCryoport Thanks for the issue!

I has been implementing multiple-file support in the studio since last week. I currently have a directories/files tree and file system in place, however there are still a few things missing that should be there and aren't, like:

• cross-referencing between files - here you have to do a proxy for cross-referencing and give the content of the files from the browser and not from the local file system (implementation details)
• showing diagnostics on each file as well as "watching" for changes to files, i.e. if you have 2 files, one application document and one common.yaml, then when you change something in common.yaml you should know if you have get errors from referenced components in the application document (from common.yaml)
• bundling files to download them to your local system - if you use references in the browser then you should also be able to download a file from the Studio with the correct references against your file system.

Of course we have a lot of features that we should implement but I will implement in next weeks (after merging basic support for multi files), like:

• moving folders/files + drag&drop for this
• prompting for possible references to be used when someone types $ref (autocompletion)
• and others very common things in e.g. Stoplight Studio etc.

I don't want to say when we will have it merged and available in the Studio but after the new year at the latest. However, I want the PR itself to open later this year. We are also starting the Christmas period and a lot of people may take time off in the next few days and I also don't know who will do the review for that PR, so please, stay tuned :)

[philCryoport]

@philCryoport Thanks for the issue!

I has been implementing multiple-file support in the studio since last week. I currently have a directories/files tree and file system in place, however there are still a few things missing that should be there and aren't, like:

* cross-referencing between files - here you have to do a proxy for cross-referencing and give the content of the files from the browser and not from the local file system (implementation details)

* showing diagnostics on each file as well as "watching" for changes to files, i.e. if you have 2 files, one application document and one `common.yaml`, then when you change something in `common.yaml` you should know if you have get errors from referenced components in the application document (from `common.yaml`)

* bundling files to download them to your local system - if you use references in the browser then you should also be able to download a file from the Studio with the correct references against your file system.

Of course we have a lot of features that we should implement but I will implement in next weeks (after merging basic support for multi files), like:

* moving folders/files + drag&drop for this

* prompting for possible references to be used when someone types `$ref` (autocompletion)

* and others very common things in e.g. Stoplight Studio etc.

I don't want to say when we will have it merged and available in the Studio but after the new year at the latest. However, I want the PR itself to open later this year. We are also starting the Christmas period and a lot of people may take time off in the next few days and I also don't know who will do the review for that PR, so please, stay tuned :)

Thank you for the update @magicmatatjahu -- I knew somebody had to be working on it!

If you want testers, please let us know. There are a few people on my team which would rather use the web-based Studio app instead of an IDE plugin.

[magicmatatjahu]

@philCryoport HI! Update from me. I will be off 23rd December - 9th January (maybe in some days I will work few hours, but have in mind that maybe) so I won't work on that feature. There is a draft PR with current code #538 and preview https://deploy-preview-538--modest-rosalind-098b67.netlify.app/

There are many more changes than I thought as there is still a lot to do. I've written a TODO list of what I still need to implement, but the basic stuff already works, such as creating folders, files, opening files in tabs, rendering current opened file etc. If you want to test it, go ahead and give feedback, but as I wrote, a lot of things are not there and it doesn't give 100% UX, so please don't waste your time. In fact, I don't know how much longer it will take, but I hope that in January it should already be permanently in the Studio - I write should, because I never know if I will have to focus on other things around AsyncAPI.

If you wish, you can test the handling of files from your filesystem - right-click in the Files arena on the left and select Open directory (only works in Chrome), then you open the folder and updating the file in Studio will automatically save the changes to your drive. Updating from filesystem->Atudio doesn't work yet because there is no native API for this at the moment so I will have to write my own based on time intervals.

Merry Christmas (if you celebrate it) and a happy new year!

[philCryoport]

@philCryoport HI! Update from me. I will be off 23rd December - 9th January (maybe in some days I will work few hours, but have in mind that maybe) so I won't work on that feature. There is a draft PR with current code #538 and preview deploy-preview-538--modest-rosalind-098b67.netlify.app

There are many more changes than I thought as there is still a lot to do. I've written a TODO list of what I still need to implement, but the basic stuff already works, such as creating folders, files, opening files in tabs, rendering current opened file etc. If you want to test it, go ahead and give feedback, but as I wrote, a lot of things are not there and it doesn't give 100% UX, so please don't waste your time. In fact, I don't know how much longer it will take, but I hope that in January it should already be permanently in the Studio - I write should, because I never know if I will have to focus on other things around AsyncAPI.

If you wish, you can test the handling of files from your filesystem - right-click in the Files arena on the left and select Open directory (only works in Chrome), then you open the folder and updating the file in Studio will automatically save the changes to your drive. Updating from filesystem->Atudio doesn't work yet because there is no native API for this at the moment so I will have to write my own based on time intervals.

Merry Christmas (if you celebrate it) and a happy new year!

Hi @magicmatatjahu thank you for putting your efforts into this. Enjoy your holiday break!

[Neverbolt]

Hi, I would love to have this feature. Since it has been a while since the last update here or in the PR I wanted to ask if there is any way to support?

[danielebra]

@magicmatatjahu looks like you have made quite significant progress to this in #538

Any plans on getting this across the line?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[philCryoport]

@magicmatatjahu looks like you have made quite significant progress to this in #538

Any plans on getting this across the line?

Looks like it was closed without merge

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[ashmit-coder]

Hey @Amzani is this still open and assignable?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[Amzani]

still relevant

[Shurtu-gal]

still relevant

[Shurtu-gal]

@Amzani is it fine if I work on this under the bounty program?

[philCryoport]

@Amzani is it fine if I work on this under the bounty program?

@magicmatatjahu apologies, where did this leave off? See @Shurtu-gal 's proposal to work on this under the bounty program.

[aeworxet]

Bounty Issue's service comment

Text labels: bounty/2024-Q4, bounty/advanced, bounty/coding
First assignment to regular contributors: 2024-09-20 00:00:00 UTC+12:00
End Of Life after: 2024-10-31 23:59:59 UTC-12:00

@asyncapi/bounty_team

The Bounty Program is not a Mentorship Program. The accepted level of Bounty Program Participants is Middle/Senior.

Regular contributors should explain in meaningful words how they are going to approach the resolution process when expressing a desire to work on this Bounty Issue.

[Shurtu-gal]

Assigning to myself.

[aeworxet]

@Shurtu-gal is an AsyncAPI Maintainer specified in https://raw.githubusercontent.com/asyncapi/community/master/MAINTAINERS.yaml, so they fall under the first category in the prioritization list.

[aeworxet]

Bounty Issue's Timeline

Complexity Level	Assignment Date (by GitHub)	Start Date (by BP Rules)	End Date (by BP Rules)	Draft PR Submission	Final PR Merge Start	Final PR Merge End
Advanced	2024-09-16	2024-10-07	2024-12-01	2024-10-27	2024-11-17	2024-12-01

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

Keep in mind the responsibility for violations of the Timeline.

[Shurtu-gal]

Just some updates from my side.

I have been trying to implement this for the the past couple of days. I tried using code from #538 however I have been facing major road blocks as the design decisions are not very clear. So, I am thinking of implementing this anew.

Meanwhile a initial set of requirements would be very helpful @philCryoport.

cc: @derberg @aeworxet @magicmatatjahu

[aeworxet]

@fmvilas
Fran, do you have any suggestions on how this functionality should be implemented?

[fmvilas]

I honestly don’t have any ideas on how to implement this in a good way. Filesystem access is strictly forbidden for browsers for security reasons. The only way to circumvent this, IIRC, is by asking the user to load the files themselves using the classic “Upload file” dialog. That could work however it’s not very convenient especially if you have many file references or you’re working on them and want the changes to propagate to Studio. IMHO, we’re probably stretching the web platform too much here. We have the VS Code extension for this purpose. It’s always going to be more convenient than the browser-based Studio application.

[fmvilas]

Another option that comes to mind is creating an agent like Postman does. It's basically a local server the user installs. Over WebSockets or HTTP, Studio will connect to the local server to search for the files in the local system. But again, it's an overkill considering we already have VS Code and IntelliJ extensions.

[Shurtu-gal]

Thanks @fmvilas

[Shurtu-gal]

Currently parser uses readFile for resolving file based references.

So I have 2 solutions:-

1. Bundle all the files into a single file and then parse it.
2. Implement virtual filesystem in browser.

Any suggestions for any other approach @fmvilas @magicmatatjahu @derberg ?

[fmvilas]

Bundle all the files into a single file and then parse it.

How? You don't have access to these files 🤔

Implement virtual filesystem in browser.

Cool but you still don't have access to the content of the files so, am I missing something?

---

A more broader question here would be: why @philCryoport needs this and why isn't it enough with VS Code and IntelliJ plugins? We might be trying to fix a problem that nobody has or that it's already solved through IDE plugins 🤷

[Shurtu-gal]

How? You don't have access to these files 🤔

There two sides basically one is live studio through CLI and another is through file editing in web app itself ( Something similar to onlineGDB and other online IDEs I suppose).

A more broader question here would be: why @philCryoport needs this and why isn't it enough with VS Code and IntelliJ plugins? We might be trying to fix a problem that nobody has or that it's already solved through IDE plugins 🤷

Yeah, this is a valid point though.

[aeworxet]

I searched a bit, and it seems like Visual Studio Code for the Web has the functionality of opening full directories.

To achieve this, it uses File System API, which is implemented in Chromium derivatives.
Can I use doesn't give a clear answer whether it's talking about exactly this API.

I was able to 'Open Folder' in browsers Chrome, Chromium, Opera (it should also be available in Edge too.)

I WAS NOT able to 'Open Folder' in Firefox and Brave (even with this fix,) but it still allows to 'Open Files' instead.

I suppose it's worth taking a look at how Visual Studio Code for the Web is built.

Nevertheless, I am also very interested in whether there is a practical application for this functionality beyond wrapping Studio into Electron and presenting it as a desktop app.

[Shurtu-gal]

As this functionality is already accessible through https://github.com/asyncapi/vs-asyncapi-preview and due to non respondence of the requester I guess this bounty can be dropped temporarily.

P.S: For context this is the slack message which started this https://asyncapi.slack.com/archives/CQVJXFNQL/p1724788352337059?thread_ts=1724788352.337059&cid=CQVJXFNQL

Let me know what you think @fmvilas @derberg @Amzani @aeworxet

[aeworxet]

Visual Studio Code for the Web doesn't recognize $refs, which indicates that Microsoft didn't consider this functionality to be of any priority for the web version of VS Code since 2021.

I propose discarding this GitHub issue completely as irrelevant to avoid polluting the backlog.

[Shurtu-gal]

I talked with @derberg and he said this feature is requested by couple of users and would be a good to have as not everyone use VS code and for that matter IDE as well.

Hence, I would have a meet with the maintainers and discuss the future they are envisioning for this project.

Meanwhile @aeworxet can this bounty be put on pause(or transferred if you prefer) for now, and we can continue once there is a clear understanding of its need.

[derberg]

yeah, looking at history of features requests, my view is that people use Studio for "sharing their AsyncAPI documents". IDE will never replace that experience. So if I was driving this project as maintainer, I would focus on features that help with this collaboration, that people can share documents (with or without references), and in future do more, and more, and maybe somebody will pick Studio up and build some commercial plugins on top, like private sharing of documents with invited people only 😉

I'm not maintainer though 😃

[Shurtu-gal]

I'm not maintainer though 😃

Your input is always valuable and appreciated. Also you could become maintainer though 😉.

[aeworxet]

Supporting clickable links to external files in $refs, even in the physical filesystem, was not an easy task even for the VS Code desktop app years ago (spoiler: it ended with nothing).