As part of our quarterly review process of all the FAQs, we have decided to touch upon the most frequently talked about subject, ETL vs EAI. In this article we look to address some of the pain points involved in the decision making process between the two, what they mean to your Business and when you should draw boundaries between these two  integration techniques.

As part of any data integration strategy, it is a golden rule of thumb to follow ‘prevent and prepare rather than repent and repair’. In most cases concerning Business Intelligence initiatives, if you know what you don’t need, you are already half way through in the quest to what you want to achieve. Business Enterprises invest millions of dollars on BI initiatives and Data Warehousing and hence rely on a consistent, accurate and reliable data. Thus following proper integration techniques becomes all the more vital. With concerned to enterprise-integration strategy, EAI is essential for maintaining a perfect harmony between business needs and IT solutions where as Data integration, a function of ETL is a constant need as inconsistent data at the heart of any BI initiative fails to provide an accurate picture of the Business. Hence the vital question EAI or ETL?

What is EAI (Enterprise Application Integration)?

EAI may be defined as a process of aligning a business’s strategic vision with its information technology. In other words EAI is mainly process integration that exists to allow client applications to operate on data from across the business through API calls without regard to its location, encapsulating technology or format. It involves integration of incompatible business applications within and beyond enterprise to allow them to talk to each other seamlessly and to share data in real time.

EAI solutions enable the automation of end-to-end business processes by coordinating sequences of tasks and resources (both systems and people) that perform them. EAI solutions support sophisticated exception management and the dynamic modification of processes even when processes are underway. EAI involves developing a ‘unified view’ of an enterprise’s business and its applications, seeing how existing applications fit into the scheme of things, and then devising ways to efficiently reuse what already exists while adding new applications and data.

EAI integration is mostly message-based, transaction-oriented, point-to-hub, brokering and transformation for application-to-application integration. Some of the core benefits offered by Enterprise Application Integration are.

1. A focus on integrating both business-level processes and data
2. A focus on reuse and distribution of business processes and data
3. A focus on simplifying application integration by reducing the amount of detailed, application specific knowledge required by the users

What is ETL (Extraction, Transformation & Loading)?

As the acronym stands, ETL is a process of extraction / reading a database, transforming or validating the data, and lastly loading / writing to a database. Extract, Tranform & Load are three database functions that are combined into one tool (like DBSync) to pull data out of source databases and place it into target databases. ETL is used to migrate data from databases to others, to form data marts and data warehouses and also to convert databases from one format or type to another.

Extract - The process of reading data from source systems. Data can be extracted in schedule-driven pull mode or event–driven push mode. Pull mode operation supports data consolidation and is typically done in batch. Push mode operation is one online by propagating data changes to target data stores.

Transform - the process of converting the extracted data from its existing form into the format it needs to be in so that it can be placed into other systems or databases. Transformation occurs by using rules or lookup tables or by combining the data with other data.

Load - The process of creation and execution of workflows to write data into the target systems. Data loading may cause a complete refresh of a target data store or may be done by updating the target destination. Interfaces here include de facto standards like ODBC, JBDC, JMS, or application interfaces.

ETL is mainly designed to process very large amounts of data, ETL provides a suitable platform for

1. Improved productivity by reuse of objects and transformations
2. Strict Methodology
3. Better Metadata Support, including impact analysis.

Drawing Boundaries for ETL & EAI

As mentioned in the above sections, both techniques rely on the concept of a unified view and the definition of a mapping that allows data from many disparate sources to be “projected” onto that view. But what differ are the purpose, speed and direction and amount of data that are transformed and placed within the unified view from the external sources.

Process-level integration mainly deals with building enterprise-wide business workflows and processes and incorporating existing applications into those processes. EAI middleware acts as the workflow engine integrating applications in near real time, passing small amounts of data through message queues and a series of stages. EAI tools provide much more complete workflow capabilities than ETL tools, which provide simple workflow.  EAI tools, and especially their workflow components, provide very sophisticated GUI development environments that enable design and management of very complex business processes. Here transformations are focused on ensuring a common understanding of the context and meaning (semantics) of the data involved within the message, the more likely a proven EAI is more appropriate.

With regards to data integration, ETL tools and also next generation ETL clearly holds an advantage whether in batch or real time. Synchronizing data between two applications involves a lot more data manipulation than simply moving data from point Source to Target. It involves data intense tasks that depend upon either RDBMS efficiencies/scalability or in-memory data caching to achieve the necessary throughput. Typically, enterprise data warehousing projects require you to move large amounts of data within relatively small windows of time. Also since ETL tools were born out of the relational database world, and thus are adept at performing SQL oriented transformations. They deal mainly with pulling data out of multiple relational tables, understanding the meaning and relationships between the tables, combining, merging, or joining that data, and augmenting it with data from their sources. Since this is more than just moving data from Source to Target, ETL is more appropriate.

As you grow into your business, you add applications. You start off with Quickbooks, then add eCommerce applications and more as you start to organize your business and life. Then you realize “Geez” I have all these data all across and I am sitting down copying data from one application to the other, reconciling payments and more. Every business we have spoken to have its own story…

While we are trying to address most of these issues, lately we have received a lot of interest in our Intacct Integration Connector. Intacct themselves provide Salesforce.com integration with a pre-built setup, works for most case and does not in others. But a bigger problem is integrating with existing in-house application like Shopping carts and others. If you look at the Intacct ecosystem, there are not many, actually none which are cost effective.

We had a similar experience with one of our prospects and we recorded our session to show you all how we can easily use DBSync to integrate with in-house databases. We have also used the same mechanism to integrate with other CRM’s like SugarCRM and vTiger. We hope you find this informative.

This is a two part YouTube video.

Part 1:

Part 2:

Please feel free to contact us if you have any questions. You can also register at www.mydbsync.com to get access to our Wiki for more examples.

Many of our customers are not use on how best to streamline business process between CRM like Salesforce.com or Microsoft CRM or eCommerce or shopping carts with accounting systems like Quickbooks, Intacct or Microsoft Dynamcs GP.

A “typical” business flow is like

Opportunity -> Estimate/Quotes  ->Sales Order -> Invoice ->Credit

Here is how they should think about it -

Do you carry inventory?

If Yes (manufacturing, retail etc)

Holding inventory has a all sort of new ounces to worry about like managing adequate Inventory levels, supply chain – remember – holding inventory is not a good idea as it locks up working capital. From a standpoint of online orders or CRM Opportunity integration, they should always use “Sales Order”. Here are some are benefits -

• Sales Order does not hit GL and usually is a “notification” for fulfillment to deliver
• As you ship or deliver goods, you can generate multiple Invoices against a Sales Order
• Sales Order will deduct from inventory but not necessarily commit shipped on Inventory. This gives you a better count of inventory on hand vs commited.

So the recommendation is

Opportunity -> Quotes -> Sales Order -> Invoice (on or more) -> Credits or RMA (Return Merchandise)

If No (Services companies, Professional Services, Consulting etc)

You deal with people most of the time and billing is usually a function of scope of work or time spent.  Normally you would have many contract terms on billing as mile stone based, retainers and others. So in your case you can have multiple setups of invoicing and would trigger at different times.

• If you receive retainers, you can apply towards an Asset account and as you invoice it, move it from the asset account to the income account
• You can use Estimate to hold the hours allocated
• Invoices (and PO or Bills on AP if you are sub contracting) can be generated from Time & Expense modules
• The same can be done if you invoice based on milestones. You can deduct hours or Unit of work against your Estimate towards invoicing

So the recommendation is

Opportunity -> Estimate (hold hours) -> Invoice (invoice on spent time on milestone and track against Estimate) -> Credits

These are some simple rule of thumb when designing your business process flow. Hope this helps our audience.

Most of us are aware Master Data Management is not just managing data, but it’s beyond just MDM, it actually requires use of technology, data integration tools & processes to create and maintain consistent and accurate list of master data in organizations. Master Data Management or MDM typically called as might become a nightmare for many organizations either large or small if it’s not planned and implemented at the right time. So how organizations need to plan for it?

3 Steps to Plan for Master Data Management

1. Identify Master Data Requirements – First and foremost thing when you have decided to go for master data management is the need to evaluate your existing systems, processes, infrastructure, organization structure and more, then try to find out if your organization actually requires an MDM and start identifying the requirements by asking questions as what’s are the business values to be achieved with this information from MDM & more.

2. Business Process Requirements – Once you have identified your MDM requirement your next immediate things to do is to start asking questions related to business process like what needs to be managed, who would be managing the MDM, what are the processes to be considered, what are the integration tools required, who can access the information, what’s the format of display, who can alter or edit the information, how to maintain the master list should it be in single format or multiple formats/copies, what locations to be maintained and more.

3. Type of Analytical requirements – Next would be to find out what types of analytical requirements for your organizational MDM plan like for areas as business intelligence & management reporting for accurate information for various roles and responsibilities.

From these steps and information start building the MDM project plan & look out for vendors like IBM, Informatica, Oracle, SAP & others who can implement MDM for your organization. During the next few years Master data management might become mandatory due to increase in mergers & acquisitions worldwide. Developing a strong MDM for large or small business helps in organizations success.

Learn more about DBSync

All major auto manufacturing companies either large or mid size have implemented JIT (Just in Time) concept for their manufacturing units across the globe. JIT was pioneered, implemented and practiced by Toyota few decades back. JIT (Just in Time) concept is nothing but ensuring inventory to be available on time for its manufacturing or assembling units. Though these manufacturing organizations use JIT concept do they need sales & inventory forecasting software as they run on JIT (Just in Time).

Four Reasons why it’s crucial:

• Accurate inventory forecasting – For organizations running on JIT system it’s very important to have its inventory to be forecast accurately to keep its manufacturing or assembling units be running on schedule or the system would get collapsed due to wrong forecasting methods.

• Critical components on JIT – Not all components can be made to run on JIT system as few of the components play a critical role in final output of the product. So it’s better to have forecast these components on higher level for usage at critical juncture with a small percentage of inventory in the system

• Collaborating with Suppliers – Suppliers supplying components or materials to JIT (Just in Time) organizations need to collaborate with vendors for accurate forecasting of components or materials and need forecasting software which works with suppliers and vendors IT infrastructure.

• After Sales Service – Although JIT (Just in Time) can be implemented for manufacturing components what about for after sales service? For parts to be used in after sales service organizations cannot implement JIT (Just in Time) as it would cripple the entire system of service and organizations image. So in order to have better after sales service support organizations need forecasting software with sufficient inventory levels for day to day operations.

JIT (Just in Time) has been implemented successfully by few major manufacturing units across the globe, but not 100% successful. So manufacturing organizations need forecasting software for forecasting inventory required for its day to day operation, critical components & components for after sales service.

For more on inventory level forecasting visit www.mydbsync.com

Most of the large & medium size manufacturing companies & retail chain stores have inventory forecasting software’s for forecasting inventory & future sales number based on past data, industry trends & economic conditions All statistical based inventory and sales forecasting models or software’s aren’t 100% accurate but have you ever thought this inventory forecasting software’s fail completely.

 3 reasons for failure:

• Negative sales – In many products based organizations sales happens during last few days of the month or year for achieving numbers. Few of these sales are called negative sales as they would be reversed back at a later date. Does your sales and inventory forecasting software accept stock reversing if yes then it failed to forecast sales accurately? So the sale happened isn’t actual sale and all forecast done for future dates in invalid due to negative sales. Do you still rely on the data?

• Promotional offers – Does your sales and inventory forecasting software consider your organizations promotional offers while forecasting sales and inventory? If yes, what’s the base for such forecasting as each promotional offer is unique to attract more and more customers and varies depending on the sales trend, market scenario and geography?

• Newly Launched products. All products have specific product life cycle from launching stage, growth stage, maturity stage and finally phase out stage. But few products are launched which wouldn’t fall into any existing product category as they have been launched to create a new product category itself like world cheapest car. In such new launches does your forecasting software forecast sales & inventory. If yes what’s the basis for such forecasting or does it considers competitors most similar products. If yes it’s not forecasting for your product but for your competitor products. Do you really need such forecasting data?

Considering these facts under which sales and inventory forecasting software’s fail, let’s assume we would soon have sales and inventory forecasting software which forecasts up-to 99.99% accurately.  Till then we need to rely on our old forecasting software’s only.

For detailed sales &inventory forecasting software visit www.mydbsync.com 

Learn more about DBSync & Lokad

If you are starting to feel that your current accounting and business solution is slowly but surely becoming inadequate, not catering to your business needs entirely and also isn’t keeping up pace with your organization growth, then you have just outgrown the entry-level accounting software. In such a scenario if you’re seeking a solution that can overcome these limitations then, Microsoft’s ERP solution,  Dynamics GP (Great Plains) is an ideal upgrade that can cater to your business needs and perhaps more.

With the advent of cloud, there has been a fundamental shift in the way businesses are being operated these days. This change is more evident than ever before as more and more organizations require both on-premise and online solutions. In most cases these processes are interconnected, leading to the integration of financial systems, supply chain management, CRM, e-selling, and other key business processes. In view of this, GP provides great flexibility for small and mid-sized businesses alike to deal with the IT challenges they face in the new interconnected economy. Being a Microsoft Certified partner, we have managed to make better use of features incorporated in GP and has enabled us to provide successful integration with host of applications.

With DBSync, Microsoft Dynamics GP users can connect to databases, on-demand CRM tools and a variety of other applications to improve productivity and data visibility. Some of the features that customers can make use of are as follows,

• Synchronize Customer Master Databases with Microsoft Dynamics GP
• Convert Opportunities Won to Order generated in MS Dynamics GP
• Convert Opportunities Won to Invoice in MS Dynamics GP
• Integrate E-commerce website orders to MS Dynamics GP
• Integrated payments received from online payment systems for reconciliation.

DBSync for Salesforce.com & Microsoft Dynamics GP integrates Salesforce objects like Accounts, Contacts, Products and Opportunities to GP as Customers, Sales Order Processing (SOP), Purchasing Order Processing (POP ), Receivables (RM), Items, Invoicing Transactions, and many more, while preserving links between tables.

Quality of data plays a major critical role in industries like retail, telecommunications & financial services with large customer base.  Most organizations falling under these industries in a rush to capture market share and to add new customers give secondary thought to quality of data captured, leading to substandard customer data with inaccurate, duplicate & outdated customer details.

Inaccurate or poor customer data is due to non standardization of data capturing process or format, multiple data capturing points across the organization & data storing in incompatible systems and formats.

Organizational impacts due to poor data quality

• Business performance – Due to duplication of data it’s difficult for marketing to cross sell other products and services, profiling and segmentation of customers leading to decrease in business performance
• Customer retention – Due to poor data quality cost of customer retention is high as it requires multiple interactions with customers to understand their actual requirements which would have been solved with  accurate data and past transaction details
• Customer Mapping – Due to multiple entries of same customer with contradicting details it would lead to wrong mapping of customer s with wrong product or service leading to valuable customer loss and revenue to organization

So organizations with large customer data should ensure their data capturing methods are standardized across all data capturing points with latest data integration software for a satisfied customer and profitable organization.

Learn more about data integration by visiting www.mydbsync.com

Learn more about DBSync

Have you ever imagined few years back that our world would see faster changes in technology front with newer and better products and features? Yes, let’s accept the fact that we are in world with fast changes in technology right from accounting software to various payment modes by customers and organizations.

All nonprofit organization are run by donations made by its members and non members in various modes of payments like cheques, credit card, debit card, mobile banking, online banking, PayPal & other modes.  Currently QuickBooks online edition accepts payments only through credit card payment by its donors, members & non members.

Reasons why QuickBooks should incorporate these features in QuickBooks online edition

• Higher ROI (Return on Investment) – All nonprofit organization either small or large look for return on investment as they rely only on donations. If these latest payments modes are automated rather than manually entering them it would reduce a substantial amount of time and money for the organization
• Lower TCO (Total cost of ownership) – Most accounting software vendors like QuickBooks and others would always reply on lower TCO for their customer to be more competitive in the market. By these features automated payments modes the cost of ownership would reduce drastically for customers

We would soon see a world without any cheque payments either by organizations or customers. To be in tandem with the changes in payment modes by customers and organizations QuickBooks need to incorporate these features as quickly as possible in QuickBooks online edition to be ahead of its competition and being a next generation company for its customers.

Compared to few years back we are now in an ever changing business world on real time basis and to this fast changing dynamic world even retail chain stores from hyper markets to smaller retail stores & manufacturing companies have implemented sales & inventory forecasting software for quick and easy forecast either web or desktop based. Most of these forecasting software’s are developed with all the modern statistical forecasting techniques with features as inventory in hand, forecasting inventory required, scalability, stock level alerts & minimum customization.

But what’s the next level of forecasting for more accurate inventory levels?

• Anticipating the unexpected – Few of the larger retail chains around the world have customized their inventory forecasting models with anticipating unexpected and alternatively  have incorporated solutions in forecasting software. But what about the same features availability for small to medium size retail stores. If yes at what cost?

• Collaborate with suppliers in sales, planning and inventory forecast – In today’s scenario it’s impossible for any manufacturer to manufacture all the items they require and the same has been outsourced to suppliers. Does your inventory planning software consider suppliers inventory too for your organizations inventory forecast? If not include them

• Include your customers in planning your sales & forecasting inventory – All types of retail stores and manufacturing organization have loyal customers with regular purchase patterns.  Can your inventory forecasting software include your customer’s future requirements by understanding their purchase patterns? If yes then your forecasting software is ahead of it’s time

All the sales & inventory forecasting software’s currently available in the market do have forecasting sales & inventory capability confined to internal organizations inputs* only. If these software’s are included with all the next level forecasting inputs at affordable prices then it would change the way forecasting in done by collaborating with partners, suppliers & customer’s on real time basis.

For more detailed sales & inventory forecasting software visit www.mydbsync.com 

Learn more about DBSync & Lokad