ETL and Encompass

There’s no way around it. Integrating any sort of software into your Enterprise that aggregates data from multiple sources means one thing: ETL. Three simple letters that inspire exasperated sighs from developers, aggravation for DBAs, and headaches for Project Management. At Perception, we understand the pain and aggravation ETL can add to the software adoption process, and we didn’t want to add further challenges to your already hard-working DBAs and IT. Thus, the Encompass DataLoader(EDL) Framework was born. Drawing upon our previous experience onboarding our customers, as well as our expertise in Intelligent Navigation, the Encompass DataLoader Framework was designed from the ground up to be:

  • Friendly
  • Flexible
  • Functional
  • Future-focused

So how does the EDL Framework satisfy that criteria?  Let’s take it one at a time:

Friendly not just to the users, but also to the admininstrators and IT staff tasked with maintaining the Encompass software suite in production. The EDL framework eliminates the need for a lengthy onboarding process by separating the data flow from the application logic via an easy-to-understand and generate configuration file. You configure which data sources you want data fromand leave the heavy lifting to us. A full suite of easy-to-use administration and troubleshooting CLI tools ship along with the framework, giving you the visibility and control you need to respond to any issues that may arise. The EDL Framework also ships with an interactive UI allowing for complete ‘self-service’ creation of the ETL process.

No off-the-shelf software can possibly account for every business’ needs 100% out of the box, and the EDL Framework was designed with this firmly in mind.  EDL ships with a wide range of Connectors for a wide range of data sources and data transformations that will suit the lion’s share of cases, but for those times where more is required, EDL supports the inclusion of customer-created Connectors and Transforms via our Groovy scripting API, or our Java development kit.  And since the EDL Framework is 100% Java based, your ETL process will work just as smoothly in Windows as it does in Linux.  Installation of the EDX framework is simple, everything you need to get started is included in one small zip file.  Simply extract the zip to your server, and you’re ready to get started.

But what if you already have an ETL tool and process that works well for you that you know and love?  We’ve got you covered.  With just a few simple tweaks to your ETL process, you can deliver an EDX package to our indexer, and we’ll take care of generating the Encompass index from there. There when you need us, out of the way when you don’t. Use as much or as little of our EDL framework as you need.

As mentioned before, the EDL Framework is equally at home on Windows servers as it is on Linux/Unix. Written on top of Java 7 with a minimum of external dependencies, the EDL Framework makes full use of multi-threading, allowing you to parallelize the ETL process not only across Extracts and Transforms, but within each individual Extract and Transform, all by making a few simple edits to the workflow configuration.

Future Focused
Data is constantly increasing in size, and the EDL Framework will continue to evolve to keep up with the challenges inherent in managing it. Our next major releases of the EDL Framework will include capabilities for Distributed and Incremental ETL, ensuring that we’re constantly delivering your engineers the most up-to-date data possible. There’s no need to worry about future releases of the EDL Framework breaking what you already have either; every major release is guaranteed to either be backward-compatible, or will provide a migration tool to ensure that your ETL process continues chugging along with a minimum of intervention.

Heard enough? Ready to take Encompass Product Navigation and Data Loader for a spin yourself? Download our demo and give it a whirl.

Leave a Reply

Your email address will not be published. Required fields are marked *