In this tutorial you will learn how to ingest data from a Mendix application. In the following steps you will learn how to configure the Mendix Data Loader component, a Snowflake application that is deployed in your Snowflake environment that ingests your Mendix data.

Mendix is a leading platform in the low-code application development domain. The data structure for the application is retrieved and the transient target tables for the data ingestion are created dynamically.

Architecture diagram

What You'll Learn

What You'll Build

Prerequisites

Mendix

Mendix stands as a leading low-code platform for developing enterprise-grade applications, offering unmatched speed, flexibility, and scalability. Mendix's seamless integration with Snowflake's enterprise data environment makes it an essential tool for building robust, data-driven applications. The platform's intuitive visual development environment accelerates the creation of complex applications, significantly reducing development time while maintaining high standards of quality and performance.

Furthermore, Mendix offers extensive customization through its rich ecosystem of components, including marketplace offerings that facilitate direct integration with Snowflake. This allows Mendix applications to easily connect, query, and visualize Snowflake data, unlocking deeper insights and driving informed decision-making across the organization.

With Mendix, data engineers can focus on what truly matters—maximizing the power of their data within Snowflake—while relying on a platform that ensures enterprise-level security, compliance, and scalability.

Deploy Your First Mendix Application

Deploying your Mendix application

About this Mendix Application

The application you just downloaded, uploaded to Mendix' version control server and deployed on a free cloud sandbox environment is a free application available on the Mendix Marketplace. Its purpose is to enable, aid and inspire its users on how to tackle integration with Snowflake from the Mendix domain. This application has some pre-installed operational data to showcase the Mendix Data Loader. The pre-installed data is about movies and reviews corresponding to the movies.

How Will the Mendix Data Loader Interact With the Mendix application?

The Mendix application has a published OData service that exposes the application data for the entities (class definitions) captioned Movie and Review which are linked to one another through an association. The OData resource for this application can be found along the following path: Showcase_DataLoader -> Resources -> Published OData -> POS_Movies.

In the OData resource, the General tab contains information about the exposed service and the entities that are exposed in the service. Each entity has an endpoint from where the values can be retrieved after authentication. In the Settings tab, the metadata endpoint contains information about the exposed data structure of the OData resource. Additional endpoints are exposed for each exposed set configured in the General tab.

The Mendix Data Loader retrieves the exposed data structure from the metadata endpoint. After which the application will start to provision the application with transient target tables. Then the application retrieves the data from the service feed for each exposed entity found in the metadata. The Mendix Data Loader is developed for the extraction and loading of Mendix data, any data transformation and integration should be performed outside the scope of the Mendix Data Loader's objects.

All the exposed data will be ingested into Snowflake. To retrieve a subset of the exposed data you can use the filter query option, for more information refer to OData's Basic Tutorial.

Should any data reside in the specified database and schema from prior ingestion jobs, this data will be lost. For ingestions of multiple sources, we recommend using the same database with a different schema.

Architecture diagram

Starting the Application

Upon starting the application, a documentation page that includes usage instructions is displayed. To start the application, click the MENDIX_DATA_LOADER hyperlink in the header.

Granting the Application Privileges

The Mendix Data Loader requires the CREATE DATABASE and EXECUTE TASK privileges to create the target database where the ingested data will be stored and execute tasks used for scheduling ingestion jobs. To that end, a modal window will request you to grant the application these privileges, click Grant Privileges to accept this request.

Next, the application requires a NETWORK RULE, SECRET and EXTERNAL ACCESS INTEGRATION objects to make the external call towards your deployed Mendix application instance. To create these objects, choose the basic credentials authentication method and fill in the ingestion configuration form of the application as follows:

Note that the Mendix Data Loader also has the option to authorize the OData call using OAuth.

Then click the Submit button and navigate to the Main tab. On the Main tab click the Generate Access Script button and copy the value of the generated field. This button uses the values from the fields to create a SQL script that a user with the ACCOUNTADMIN role needs to execute. This script will create the objects in the database titled mx_data_loader_secrets and grant the application privileges to access those objects.

Access script

Open a new browser tab, log into the same Snowflake environment and create a new SQL worksheet. In here, paste the copied SQL script and hit the CTRL + SHIFT + ENTER buttons on your machine simultaneously. The execution of commands may take a few moments and should result into a table with a single column captioned status and one row with the status value "Statement executed successfully." should appear.

You have now successfully granted the Mendix Data Loader with the privileges and objects it needs to ingest data from your specified endpoint.

Start a Single Ingestion

Move back to the initial browser tab in which you had opened the Mendix Data Loader. If the input fields have the same values as the ones that you specified for the access script generation, you can now click the Ingest Data button. If not, go back one step and specify the values in the form of the application again and then click the Ingest Data button.

Creating a Task to Schedule Ingestion Jobs

Navigate to the Schedule Task tab and fill in the task configuration. The following fields are mandatory:

The other task configuration fields are optional:

Congratulations! You've successfully installed Mendix Data Loader app and moved data from Mendix application onto Snowflake.

What You Learned

Related Resources