In this tutorial you will learn how to ingest data from a Mendix application. In the following steps you will learn how to configure the Mendix Data Loader component, a Snowflake application that is deployed in your Snowflake environment that ingests your Mendix data.

Mendix is a leading platform in the low-code application development domain. The data structure for the application is retrieved and the transient target tables for the data ingestion are created dynamically.

Architecture diagram

What You'll Learn

What You'll Build

Prerequisites

If you already have a Mendix application and just want to connect to that application you can skip to step 3.

Mendix

Mendix stands as a leading low-code platform for developing enterprise-grade applications, offering unmatched speed, flexibility, and scalability. Mendix's seamless integration with Snowflake's enterprise data environment makes it an essential tool for building robust, data-driven applications. The platform's intuitive visual development environment accelerates the creation of complex applications, significantly reducing development time while maintaining high standards of quality and performance.

Furthermore, Mendix offers extensive customization through its rich ecosystem of components, including marketplace offerings that facilitate direct integration with Snowflake. This allows Mendix applications to easily connect, query, and visualize Snowflake data, unlocking deeper insights and driving informed decision-making across the organization.

With Mendix, data engineers can focus on what truly matters—maximizing the power of their data within Snowflake—while relying on a platform that ensures enterprise-level security, compliance, and scalability.

Deploy Your First Mendix Application

Deploying your Mendix application

About this Mendix Application

The application you just downloaded, uploaded to Mendix' version control server and deployed on a free cloud sandbox environment is a free application available on the Mendix Marketplace. Its purpose is to enable, aid and inspire its users on how to tackle integration with Snowflake from the Mendix domain. This application has some pre-installed operational data to showcase the Mendix Data Loader. The pre-installed data is about movies and reviews corresponding to the movies.

How Will the Mendix Data Loader Interact With the Mendix application?

The Mendix application has a published OData service that exposes the application data for the entities (class definitions) captioned Movie and Review which are linked to one another through an association. The OData resource for this application can be found along the following path: Showcase_DataLoader -> Resources -> Published OData -> POS_Movies. If you are planning to use your own custom application for this quickstart you will use the exposed OData service that you have set up yourself.

In the OData resource, the General tab contains information about the exposed service and the entities that are exposed in the service. Each entity has an endpoint from where the values can be retrieved after authentication. In the Settings tab, the metadata endpoint contains information about the exposed data structure of the OData resource. Additional endpoints are exposed for each exposed set configured in the General tab.

The Mendix Data Loader retrieves the exposed data structure from the metadata endpoint. After which the application will start to provision the application with transient target tables. Then the application retrieves the data from the service feed for each exposed entity found in the metadata. The Mendix Data Loader is developed for the extraction and loading of Mendix data, any data transformation and integration should be performed outside the scope of the Mendix Data Loader's objects.

All the exposed data will be ingested into Snowflake. To retrieve a subset of the exposed data you can use the filter query option, for more information refer to OData's Basic Tutorial.

Should any data reside in the specified database and schema from prior ingestion jobs, this data will be lost. For ingestions of multiple sources, we recommend using the same database with a different schema.

Architecture diagram

Starting the Application

Upon starting the application, a documentation page that includes usage instructions is displayed. To start the application, click the MENDIX_DATA_LOADER hyperlink in the header.

Creating Data sources

The home page, titled Data Source Overview, displays all your existing data sources. To create a new one, click the Create button to access the Create New Data Source page. Here, you will provide the following information:

When you press Save for the first time, a popup will appear requesting permissions to grant the application the following privileges:

  1. CREATE DATABASE: To create the target database where ingested data will be stored.
  2. EXECUTE TASK: To enable the scheduling of ingestion jobs.

In the modal window, click Grant Privileges to approve the request. Once privileges are granted, press Save again to save your data source. You will then be redirected to its details page.

Configuring Authentication

To set up authentication, navigate to the Authentication Configuration tab and click the Edit button to open the Edit Authentication Config page. To start, there are two options, Basic credential authentication and OAuth. For this quickstart we will guide you through the basic credentials authentication configuration.

Choose the Basic option in the presented dropdown field and specify the following fields:

Configure the Staging Area

The final step before data ingestion is to configure the staging area. The staging area consists of a target database and target schema where data will temporarily reside before you transport/process it into a database owned by your organization.

Architecture diagram

Start a Single Ingestion

Navigate to the Data Ingestion tab ands click the Ingest Now Button. When the ingestion has finished you are presented with success or error message.

Architecture diagram

Creating a Task to Schedule Ingestion Jobs

Navigate to the Data Ingestion tab and click the New Task button to start configuring a task for scheduling ingestion jobs. The following fields are mandatory:

The other task configuration fields are optional:

Now press the Schedule Ingestion Task button. The first time you configure a task you will be prompted to grant the USAGE permission on a warehouse you want to use for all tasks created by the Mendix Data Loader. You can view details of the created task on the Data Ingestion tab where you can also view its performed ingestion jobs, suspend/enable the task, and drop the task.

Congratulations! You've successfully installed Mendix Data Loader app and moved data from Mendix application onto Snowflake.

What You Learned

Related Resources