Microsoft releases version 1.0 of .NET for Apache Spark. The open source package is intended to connect .NET development with the platform for big data processing.
Apache Spark scales complex data analysis and machine learning applications across a cluster. The software has established itself as a standard tool for the analysis of large amounts of data and is written in the Scala programming language.
Support for .NET applications
In addition to support for .NET applications for .NET Standard 2.0 or older, the current release also offers the option of using user-defined functions from .NET (UDFs for short).
In addition, users get access to the Apache Spark DataFrame APIs (versions 2.3, 2.4 and 3.0) and the ability to use Spark SQL – a module from Apache Spark for processing and querying structured data.
The big data engine is currently in the third major version. Further information on .NET for Apache Spark can be found in the release notes and in Post on Microsoft’s .NET blog. There you will also find detailed instructions on how developers can start their projects.