Data Managed
Data Integration
Simplifies, Automates &
Accelerates ETL

Simplicity: Our Data Dictionary Manages
Enterprise ETL Functionality

Automation: Schema Changes
Automatically Detected and Managed

Acceleration: Our Wizards Quickly Converts Your Data Systems Into Our Data Dictionary

​Our Agile Integration Reduces
ETL Management Costs by 80%

Data Managed
Data Integration
Simplifies, Automates & Accelerates ETL

Simplicity: Our Data Dictionary Manages Enterprise ETL Functionality 

Automation: Schema Changes Automatically Detected and Managed

Acceleration: Our Wizards Quickly Converts Your Data Systems Into Our Data Dictionary

Our Agile Integration Reduces ETL Management Costs by 80%

"JsonEDI manages a 1/2 million ETL data points in 5,000 tables that services 3 of the 4 largest banks in the US. Source OLTP data is a hybrid of relational and semi-structured data. This is managed with 1 ETL DBA and 1.5 ETL developers. While most schema changes are automated, the remaining schema changes are managed in an Excel file signed off by client ETL teams. This file is easily imported into JsonEDI's data dictionary and synchronized to physical schema."

-Equator.com       
       
Managing ETL Programming is Painful
Data integration is fundamentally a data issue that should be resolved with a data methodology. But traditional ETL software is a computer programming tool to create custom ETL code. ​Coding is a management headache with expensive overhead and complex coordination with the business and technology. Plus the code is inherently unreliable because it is hard coded and tightly binds to database schema, any environmental changes causes it to break. This is even more problematic as data architecture is incorporating NoSQL databases with dynamic, hierarchical schemas. JsonEDI's dynamic data approach to ETL resolves these issues and improves nearly every touch point of data integration including data architecture itself.
Big Data to Everyday ETL
JsonEDI's Java framework and use of Json documents allows flexibility in integrating with nearly any data connector. We have powerful functionality for SQL and NoSQL database integration. We have an interesting dynamic flat file management solution. Our REST based services allows streaming and near real time integration. This is ideal for integrating a search engine like Elasticsearch or as a consumer on a enterprise service bus. Multi-threading and node/partition write buffering provide high throughput to Hadoop and massively parallel processing databases. Data managed ETL has several value propositions to each of these use cases.

No ETL
Staff ?

Data Integration as a Service

Hosted JsonEDI in the Cloud or On Your Premises

Professional ETL and DBA Staff On Demand

24 Hour Monitoring and Support

Fixed Monthly Cost

Synopsis of

JsonEDI

JsonEDI provides automation to ETL similar to what WordPress provides in creating a website. First choose plugins based on what you're trying to accomplish. These plugins are rules engines for data warehousing or loading a search engine etc. Then our user wizards guide the developer by extracting information from several locations including source data, schema and data models to populate a data dictionary style repository. The metadata can be further modified, if needed, with simple tags to implement ETL functionality. Minimal or no coding is required. JsonEDI's metadata is centrally managed, responsive to schema changes, semi or fully automatically created, synchronized with database schema and is reused to automate DBA, QA and requirements management tasks. We use Json documents for internal processing and can codelessly transform between hierarchical and tabular data. 

New Concepts in ETL

Metadata Based ETL
Live, dynamic metadata, similar to a data dictionary, invokes a rich feature set of ETL functionality using simple tags. Design wizards quickly populates metadata based on data models, parsing of data or ERD tools. JsonEDI can also detect and process real time schema changes.
ETL Life Cycle Automation
It's not just faster development, our metadata technology provides a complete tool set to automate DBA, QA and business requirements tasks. Expect 80% automation across the life cycle. These are some of the many advantages to managing your data tier with data vs ETL code.
Built In ETL Best Practices
We've implemented all the complex ETL concepts using industry best practices. Many are automatic or are invoked with metadata tags without needing a highly skilled programmer. We would argue that metadata managed ETL is itself a best practice.
Enterprise Visibility
Our data dictionary can be maintained in a central repository we call Master Schema Management. This allows complete visibility and change management in both the data tier and across to the application tier.
SQL/NoSQL Integration
Powerful feature set for SQL and NoSQL integration including real time schema processing. Automatic normalization of hierarchical NoSQL source to SQL destination is ideal for data warehousing or an ODS. Or the reverse, SQL to Json to load a search engine.
We Span the Enterprise
From everyday ETL tasks to complex data models in data warehousing, Big Data or ERP integration. Batch or streaming. We integrate data between any endpoint including applications, search engines, files, REST or as a ESB consumer.

5 Keys to Our Technology

Json
Processing

JsonEDI, as it name implies, uses Json Documents for internal processing. Source data is converted to Json if needed. During data persistence, the Json is converted to the native data format of the destination. We have several reasons for using Json;
    1. We are ETL for hierarchical data first, adding relational data was easy
    2. Lightweight and fast
    3. Easy to manipulate
    4. Everyone is familiar with it

Declarative
Programming
Paradigm

Declarative programming separates the "what needs to be done" from the "how to do it". This contrasts to the imperative style of object oriented and procedural programming that explicitly defines everything in code. We placed the "what needs to be done" into our data dictionary with tags. The technical implementation, the "how to implement ETL" is already coded using ETL best practices and is now behind the scenes. The fact that the data dictionary is in a metadata repository sets the foundation for our other innovations.

Data
Managed
ETL

We maintain the data dictionary in a metadata database. This metadata is a combination of the data model, schema and ETL requirements. Using our declarative ETL we were able to keep the data dictionary simple. We support ad hoc SQL for mass updates to the metadata. The metadata can also be managed as a master data management solution, something we call Master Schema Management. This allows application developers to synchronize with data integration. There are countless value propositions to data managed ETL.

Responsive
Dynamic
Application

Since we're data managed with metadata you can choose between a design time ETL designer wizards or a dynamically managed, rules based ETL server. These concepts work well together. Start with you initial design, then as new schema changes occur at the source they can be automatically added to the data dictionary. Schema changes can also be applied to the destination.

Java
Framework

Our foundation is a Java Framework with plugins added to implement ETL. You choose plugins based on the rules you want to implement.  These could be NoSQL to SQL, data warehousing, dimensional modeling etc. The metadata completes the ETL configuration. The frame work supplies a multi-threaded ETL engine plus management of metadata, job/batch, schema, errors and cache. Our Java technology can leverage nearly any Java library to connect to almost any data connection.

See Us at SQL PASS Summit

Nov 6 - 9th 2018 Seattle, WA
Washington State Convention Center
Technical Features
Most Functionality Either Automatic or Metadata Invoked

Data Transforms

Lookups (SQL, REST or Cached)
Pivot/UnPivot
Split (Several Methods)
Merge or Union
Aggregates
Filters

Data Modeling

Primary Keys/Foreign Keys
Surrogate Key Management
Dimensional Modeling
Master Data Management Integration
External Data Integration

Automatic Data Cleansing

Primitive Datatype Cleansing
Form Value Cleansing (Address, Email etc)
External Data Quality Integration
Bad Data Error Logging
Master Schema Management

Data Warehouse/Big Data

Node/Partition Aware
Multithreaded
Write Buffering per Node/Partition
Isolation of Read/Transform/Write for Scalability
Integrate Hadoop Java Libraries

DBA & Operations

Standard Job Management
Restartable Jobs
Incremental/Full Load Job Integration
Destination Error Automatic Retry
Detailed Job Monitoring/Reporting

QA/Data Quality Tools

Record Count Reporting
Schema Compares
Data Compares Using Metadata
Detailed Error Logging/Reporting

Extensible

Compute Columns or Json Elements
User Defined Functions
Extensible Java Framework for ETL
Metadata Extendable

NoSQL Integration

Metadata Supports Hierarchical Data
Automatic Detection of Schema Changes
Automatic Normalization of Semistructured Data
Automatic Datatype Detection

Government Compliance

Meet HIPAA, EU, GSA PII Requirements
Tracking/Audit of PII Data
Remove or Obfuscate PII Data
Data Dictionary Report for Documentation

About / Contact Us

JsonEDI is based in Richardson, Texas just north of Dallas. We also maintain offices in Kochi, India. JsonEDI was founded and created by Fred Zimmerman who has 20 plus years of experience as a Enterprise Data Architect in Fortune 500 companies.
+1 (972) 813-9630

Fred Zimmerman
Founder
Udai Krishnan
Data Technology Lead

Frequently Asked Questions

How can simple tags invoke "advanced" ETL?

If you gave 10 ETL programmers identical non-trivial requirements you would get back 10 different  ETL coding packages. JsonEDI has simply prebuilt all the technical functionality. All the subjective programming decisions has been replaced with ETL best practices.

Is JsonEDI fast?

Some people assume metadata managed ETL must be slow. In fact JsonEDI is very lightweight and fast. We are not an "interpretive" technology but 100% fully compiled Java code after start. We are memory and CPU efficient using only lightweight Json documents during data processing. JsonEDI is fully multithreaded.

What is Declarative ETL?

Our core innovation is that ETL can implemented using the  Declarative Programming Paradigm. We separated the technology of "how to implement ETL" from the our data dictionary that states "what ETL to do". We also discovered this approach allows for the coordination of database schema with ETL.

Articles

Declarative Json ETL Cost Savings
Legacy ETL solutions are a procedural programming exercise. An ETL process starts with a source query pulling data, then through[...]
Top 15 reasons to use a Json ETL tool
Advantages of using declarative json ETL software JsonEDI is a powerful enterprise declarative ETL tool based on internal Json documents.[...]
Use Case: Automatic ETL schema generation
Semistructured source data with automatic ETL schema generation at SQL destination Problem: A Software as a Service (SaaS) company stores OLTP[...]
Use Case: Load Elasticsearch ETL from SQL
Multi-table SQL data merged into hierarchical Json document using Elasticsearch ETL Nearly all OLTP centric database vendors (both SQL or[...]
Data Lake vs Integrated Data Warehouse ETL
Data Warehouse ETL vs Data Lake ETL JsonEDI makes a fundamental change of the cost calculus between a Data Lake[...]
TOP