Publish delta-dotnet to NuGet

[MrPowers]

There is a delta-net repo with C# bindings for Delta Lake thanks to excellent work by @johnsusi. This library is published on NuGet as DeltaLake.

We'd like to publish this delta-dotnet on NuGet, so it's more accessible. Would DeltaDotNet be a good NuGet name? Any other suggestions?

[johnsusi]

I have no issues with handing over the nuget name "DeltaLake" to this project if it helps. It would be in line with the python version. Delta is a great technology and I like to help as much as I can to see it adopted.

Will probably keep my project around just to have a pure .net library to interact with the delta table log but since it is a more narrow use case I can switch names.

[mdrakiburrahman]

DeltaLake as the Nuget package name would line up with the project namespace in this delta-dotnet repo - if it's possible to repurpose the name from @johnsusi - but starting from a clean package history to avoid end-user confusions (if they downloaded an old version - e.g. 0.1.0-rc.8, they'd get a completely different interface).

If that's painful, perhaps Delta.Net - or DeltaLake.Net (I'm not sure what value the Lake part adds 🙂- you don't need a Data Lake to write Delta files, can do it on your local hard drive too) - similar to Parquet.Net - a successful library - could be a good Nuget name.

Since Delta.Net visually lines up with Parquet.Net, IMO it might increase user adoption too.

[rtyler]

I think the user confusion point is a good one to bring up!

The symmetry with Parquet.Net is also compelling but Parquet.Net is not an Apache Software Foundation project, and the author does a really good example of respecting their trademark by consistently referring to Apache Parquet in the readme, etc. Were we to publish Delta.Net,l that would be fine, but a DeltaLake package existing would likely lead to ongoing user confusion as time progresses.

What cause a little bit of a stinker here is that The Linux Foundation has a trademark for "Delta Lake" so there would always be a risk for @johnsusi that some LF lawyer notices and has to raise the issue to Microsoft I say "has to" because the LF lawyer(s) wouldn't want to mess with @johnsusi but the unfortunate part of America trademark law is that allowing the dilution of the trademark can cause the LF to lose the trademark. (I am not a lawyer, but I have been heavily involved in trademark issues in open source before).

Based on my read of the situation as a delta-rs nerd, assuming @johnsusi is amenable to it:

• The existing DeltaLake package republishes as Delta.Net, frankly I like this name for that purpose more since it's a pure .NET implementation.
• ⏳ we let some time pass
• delta-dotnet renames the repo to deltalake-dotnet or something like that to avoid confusion.
• delta-dotnet publishes as DeltaLake with an incompatible version number, e.g. 0.2.0 or something like that.

There may be some short-term hiccups and confusion, but I think the switcheroo would prevent any future issues

[MrPowers]

@rtyler - thanks for the comments!

I think ideally we would have a solution that would allow us to publish delta-dotnet ASAP, cause it's a blocker...

So the publishing to Delta.Net would allow us to move this one forward, but you're raising a bunch of valid points.

[dennyglee]

Could you clarify what is it that's being blocked?

The concern @rtyler raises here is not trivial. As delta-dotnet aka deltalake-dotnet gains popularity with even more production systems, we need to be diligent about avoiding confusion. This is both from reducing confusion as well as reducing/removing trademark concerns.

I'd also like to ask @johnsusi his thoughts about this approach on the matter.

[johnsusi]

As stated earlier, I will support this project in whichever way you want to go.

At the time of creating my library I was unaware of delta-incubator and the delta-dotnet project, but my plan was always to offer to donate my implementation to delta once it had enough features and stable enough. When discovering this project, I paused my effort.

Currently, my library is just a release candidate so technically we are not breaking anything by switching implementation. It will be an inconvenience of course but probably not a big one.

I see two paths for myself. If we are fine with me keeping the current naming and nuget, I'll continue supporting that as an unofficial implementation of the open delta standard, just with a different API, otherwise I will rebrand it to something else and keep it as a niche library.

Hope this clears any hinders for you on my part.

[mdrakiburrahman]

Nuget allows you to unlist packages.

John will have a very tough time ahead keeping plain old C# up to date with new Delta protocols, it's extremely hard.

Delta moves pretty fast, and high performance read/write throughput while keeping up with the latest transaction contract is a must in this domain.

I can't imagine how hard it'll be as a solo engineer to keep a niche C# Delta reader/writer project alive, regression tested, SDL friendly for CVEs, performant, bug-free, and, and ... (the list goes on).

So if John isn't planning on keeping that Nuget alive, perhaps we just pull the Nuget off and nip the lawyer problems Tyler raised in the bud. We have bigger technical problems ahead and the last thing we need in this baby-sized Delta C# user community is a naming war 😊

https://learn.microsoft.com/en-us/nuget/nuget-org/policies/deleting-packages

And push the Nuget out of this delta-incubator git repo into the John's previously claimed namespace.

Then we can all put our C# heads together and make this delta-incubator/delta-dotnet project robust, and push delta-kernel-rs to solve problems it currently has (like lack of writes, complex FFI surface area with a gazillion pointers to manage state etc etc), and do it all in a maintainable manner where this Nuget package takes off, and solves a business problem that the whole Dotnet community can benefit from by increasing Delta adoption.