Home

DIPREN is a distributed processing framework implemented on the .NET 6 platform. The framework allows vast data sets to be processed in a distributed manner. Once a processing job is started, any number of processing nodes can be added or removed until the job completes. The framework ensures that each element in the data set is only processed once.

While DIPREN is headless, it does require a database to keep track of jobs and their states. Supported are Microsoft SQL Server and Postgres, but it is fairly simple to implement support for other databases.

The data to be processed can be in any form and can be hosted in any type of database (including the file system). The only criteria is that each element in the set is uniquely identifiable. Duplicate elements are not supported out of the box.