User:Ijam duo/data replication
Contents
Replication Module
Welcome to and thanks for your interest in this module, let's share the knowledge about this replication module how it works and done :),I'am written this topic according to the SUN tutorial on the JMS from SUN website.
JMS (JAVA MESSAGE SERVICE)
OVERVIEW
As I mention earlier the core of data replication module for adempiere to make it come to live is by using the JMS (Java Massage Service) which is interesting method in J2EE. This method used a messaging method to communicate between either software or applications. This method used peer-to-peer facility: A messaging client can send messages to, and receive messages from, any other client. Each client connects to a messaging agent that provides facilities for creating, sending, receiving, and reading messages.
Messaging enables distributed communication that is loosely coupled. It is very simple and basic concept which is a component sends a message one recipient and the recipient can retrieve the massage from the sender and the best of all sender and receiver not need to be available at the same time not only that the sender does not need to know a detail about it reciver the only metter here is what is the format and destination? Therefore this would gave more flexiblity because it loose-coupled and easier to set up.
In Order to used this JMS, JDK 1.3.1 provide the JMS API(JAVA Message Service Application Programming Interface) but firstly we need to be familiar with JAVA programming, What is JMS API? Let we go through this shall we :)
You can fine more detail on - http://java.sun.com/products/jms/tutorial
Data Replication Module for Adempiere
Replication is a powerful tool in nowadays business trend. It will be wonderful if this tools can be part of adempiere, thanks to trifon that make it come true. The data replication is would be released it is a matter of time. Good Job Trifon. I interested with your concept tu used JMS(Java Messaging service) as an engine to make this replication come to life.
Replication Overview
- Replication is a process of copying the exact attribute of an object or instance. Data replication is a process of sharing information by copying the one data from one node to another and at the end all nodes will receive and store the same data from its origin. Data replication is used to ensure the consistency between redundant resources, to improve reliability/accessibility.
- Data replication is nowadays trend for business especially a worldwide company where it have many franchise company for all over the world. Reliable, Availability and consistency are 3 strong words which keep this company survive in a cruel world. Reliable data the availability data and the consistency of data is important to make them ahead from their competitor because this 3 word would affect the decision making of top managerial and to plan in a future by estimating what would happen from the figure they have.
- Data replication also give advantages whereby the data would be replicated to a multiple node and in case something happen such as disaster the important data can still be recovery and can avoid from bigger loss then they already had, beside that the updated data will be always available since all node will receive same data therefore the information received is reliable and guarantee to be as same as it central node.
- There is many technique used in data replication, because if the replication is not manage in a good manner it could loading the network and make the network become slow and perhaps would cause lost connection with others nodes this is because the data is being copied among itself within one network. There is many technique used in data replication today mostly done synchronize or asynchronize between its nodes. The common architectures used in data replication is Distribution, Consolidation, Data off-loading, workflow and Multi-site data sharing this architectures were implement in Oracle 7 replication.
- Distribution. A typical example is that of a pricing list maintained by a company's headquarters and distributed to field offices. Usually the primary owner of this kind of information is the headquarters, and field offices use the information in a read-only manner.
- Consolidation. Information on sales figures may be owned, entered and maintained at the local sales office level. This data may also be partitioned so that each office only sees data related to their own sales. A headquarters office may require full read-only access to sales from all offices in order to make forecasts or to anticipate demand. Information Off-loading. In some companies, it may be desirable to reduce processing conflicts (in terms of system resources) between data entry activity and data analysis activity. An example may be an accounting system with a high transaction rate that also serves as a management reporting system. With replication, the data could be owned and maintained by the transaction system and replicated to another database location that could be specially tailored for read-only management reporting. Disaster Site Maintenance. With full data replication, it becomes possible to maintain an accessible backup site for database operations. The “backup” site is fully accessible Strategies and Techniques for Using Oracle7 Replication 2 Part A34042 and can be used to test read-only functions, or provide extra processing capability to deal with sporadic user demands. In the event of a system failure, the backup location will queue up changes to the failed system and automatically forward them after recovery is complete. All of the prior examples employ a static, exclusive ownership model (often referred to as primary site ownership). An example of exclusive ownership where the data owner may change over time is best illustrated with a work flow example. This example illustrates dynamic exclusive ownership.
- Work flow. A work flow model illustrates the concept of dynamic exclusive ownership by passing ownership of data from database location to database location within a replicated environment. An example is a combined order entry and shipping system that places orders and assigns ownership based on the “state” of an order (entered, approved, picked, shipped, etc.). Another example is an employee expenses reimbursement system in which expenses are routed to people and departments as part of an approval process.A replication work flow environment is extremely fault tolerant, allowing different functions along a flow to proceed independently in the event of failure at locations dedicated to other functions. All of the above models assume exclusive ownership of data. However, there are several applications that can benefit from some kind of shared ownership, update-anywhere model.
- Multi-site Data Sharing. Some applications have a business practice that allows multiple locations and users to update or change information related to a single piece of data. Examples might include reservation information (at a front desk or airline counter and via a central phone number) or any data being worked on by a team (such as a problem report being worked on by a team of support analysts and technicians, or a global sales licensing deal with sales representatives from many countries adding terms and conditions). Shared ownership models must be aware of the possibility for conflicting simultaneous access that may result in data inconsistency (data conflicts). Sophisticated replication environments must be able to handle conflicts as they occur, and they should provide a declarative way to describe the methods needed to resolve them. Understanding the business rules around how to handle simultaneous conflicting access to data is crucial to any shared ownership implementation. Oracle’s replication options can enable the implementation of all of these scenarios.
Adempiere data replication
I quite stunt that Adempiere is about to have it own data replication module, This could gave advantages since the adempiere is one of the ERP/CRM support software which nowadays used widely all over the world, by implementing data replication in this software could give many advantages to many company and make it powerful tools also since data replication is a trend in future business trend.
Data replication module which done by trifon is divide it node to two major nodes which is Central Node and Remote nodes. Each of these nodes has it own function and character. Such as:
Central Node, Maintains and creates key data that are required by a remote node, usually these data are master data. But this also requires the consolidation of information generated by the remote nodes. E.g.
- Products
- Business Partners
- Prices Lists
- Payment Terms
Remote Node, is an entity that can work independently , so that the entity need to continue operation even if the central node is not available. Usually a Remote node generate operation data. but need the data that is generated and maintained in the Central Node. E.g.
- Business Partner
- POS Orders
- Shipment
- Invoice
- Payments
Business Case
Overview: This is a Company that have a headquarter with 38 store that are distribute in all country
Business Requirements: The Company need have the operation control of all your 38 store and consolidate then information Business Process to HQ
- Product Management
- Price and Promotion Management
- Purchased Management
- Material Management
- Account Receivables
- Account Payables
- General Ledger to get Income Statements
Business Process to Store
- Sales Management
- Shipment Management
- Invoice Management
- Payment Management
- Inventory Management
Replication Cases
- Replicate Master Data
- A Product is created in the Central Node and is sent to every the Remote Nodes
- A Business Partner new is created in the Remote Node and is sent to the Central Node, the Central Node sent again The Business Partner to every the Remote Nodes
- Replicate Operation Data
- A Product is sold in the the Remote Node the Order , Shipment , Invoice and Payments are sent to the Central Node
Replication Schema
Trifon used JMS(Java Message Service) method to deliver the process of the replication, the concept of this method is it the data is send via message in the XML file form. Before that The basic concept of this scheme as shown below:
Local Producer --> to Local JMS -> Remote JMS Server -> Local Consumer
- Central Node to Remote Node
- The master data is create on the central known as Local Producer.
- When the master data is created and replication is about to be processed the master data then will be export first using the export processor and this data will be transform into XML file
- Then this XML file will be send and store in JMS local server
- After the file has been stored this file will be send to the Remote JMS Server(Client JMS)
- This file will be process using the import processor before it master data is replicated into the remote node
- Remote Node to Central Node
- Data is create on the remote but this time this remote node will be Local Producer.
- When the data is created and replication is about to be processed the data then will be export using the export processor at the remote node.
- Then XML file created and will be send and store in JMS local server(Remote Node)
- After the file has been stored this file will be send to the Remote JMS Server(Remote JMS)
- This file will be process using the import processor at the Central node before it data is replicated
- Remote to remote
- Data is create on the remote but this time this correspondent remote node will be Local Producer.
- When the data is created and replication is about to be processed the data then will be export using the export processor at the correspondent remote node.
- Then XML file created and will be send and store in JMS local server(Correspondent Remote Node)
- After the file has been stored this file will be send to the Remote JMS Server(Remote JMS)
- This file will be process using the import processor at the other remote node before it data is replicated
As we can se before the data is send or receive among the nodes, the data will be store first, so this if something happen in the middle of the connection between this nodes such connection lost, the data could be recover back as soon the connection is online again.
Beside that the data also can recieved the exact data as the data send by the sender. Therefore the data is reliable and can be update regularly either from remote nodes or central node, therefore the problem of redundency or corrupted data will be overcome the top managerial could do their job and make decision while other staff input the data the only error should be concern of is from human factor itself. Beside that it also convenient from using the importLoader from Adempiere.Because when we used the importloader it only would create a file but we need to distribute manually to other nodes the import back to the server everything is done manually, as we human sometimes we forget to do so or sometime we are to lazy to do that :).Therefore it this could save time and the work become faster and more convinent.
More detail about the Replication Module
An interview with the data replication module in Adempiere
- iJaNg Sarjan: hi trifon
- iJaNg Sarjan: wassup
- Trifon Trifonov: hi
- iJaNg Sarjan: I understand that u are currently developing the replication module
- Trifon Trifonov: yes, right.
- iJaNg Sarjan: I have post some comment before this under ijam_duo
- iJaNg Sarjan: and I would like to test the module
- iJaNg Sarjan: how can I help u
- Trifon Trifonov: do yo have Adempiere sources setuped in eclipse IDE?
- Trifon Trifonov: which version of Adempiere?
- iJaNg Sarjan: i'am using the 3.3.1 version
- iJaNg Sarjan: u mean at the adempiere_trunk?
- Trifon Trifonov: ok. at the moment replication works against 3.2.0 so i need to migrate it to work with 3.3.1
- iJaNg Sarjan: ok then when u had migrate it could u let me know
- Trifon Trifonov: i will post in forums and wil send you message.
- iJaNg Sarjan: ok thanks
- iJaNg Sarjan: after that what should I do
- iJaNg Sarjan: I follow some link given in the forum
- Trifon Trifonov: yes.
- iJaNg Sarjan: on of it show how to setup this module
- iJaNg Sarjan: in swf files
- iJaNg Sarjan: but it not clear
- iJaNg Sarjan: I can't bearly read the word
- Trifon Trifonov: swf is not for replication.
- Trifon Trifonov: only readme.txt describes how to integrate replication with adempiere.
- iJaNg Sarjan: ok ok..
- iJaNg Sarjan: then the swf which u included is for what?
- Trifon Trifonov: it is as general descirption
- iJaNg Sarjan: o.i.c
- iJaNg Sarjan: thanks
- iJaNg Sarjan: do u read my suggestion in that thread
- iJaNg Sarjan: what do u think?
- Trifon Trifonov: suggestion for p2p?
- iJaNg Sarjan: yup
- Trifon Trifonov: yes but i had no time to answer.
- Trifon Trifonov: it was good suggestion.
- Trifon Trifonov: JMS supports different models.
- Trifon Trifonov: p2p, Cleint/Server and Hub
- Trifon Trifonov: so it depends how Adempiere and JMS servers are setuped in order to get proper architecture.
- iJaNg Sarjan: i.c
- iJaNg Sarjan: but it can communicated with other remote processor right?
- Trifon Trifonov: http://activemq.apache.org/topologies.html
- Trifon Trifonov: if you are interested you could read here.
- Trifon Trifonov: yes you can setup by such way that messages are transfered from remote to remote without passing to *central server.
- You have sent 1 file to Trifon Trifonov.
- Diagram.doc
- iJaNg Sarjan: what do u thing?
- iJaNg Sarjan: instead of central processor to replicated to the other client
- Trifon Trifonov: i think that it is more work than we need.
- Trifon Trifonov: but also depends what you want to achieve.
- Trifon Trifonov: usualy developers try to avoid M:M connections as it is hard to manage.
- Trifon Trifonov: also with M:M one mssage will be imported M time and this will load Server.
- Trifon Trifonov: but this topology is possible.
- Trifon Trifonov: just need to define properly JMS.
- iJaNg Sarjan: i.c now
- iJaNg Sarjan: thanks
- Trifon Trifonov: you are welcome
- iJaNg Sarjan: just wanna give some idea
- iJaNg Sarjan:
- iJaNg Sarjan: but is it possible if the central bypass to send the massage and let the remote client to do all of the job
- Trifon Trifonov: yes it is possible.
- iJaNg Sarjan: therefore the central could focus to others client
- Trifon Trifonov: Remote JMS Server can send messages to all other Remote JMS Server without passing thro Central JMS Server
- iJaNg Sarjan: yup I've read it in the wikipidia
- iJaNg Sarjan: just to get more idea on JMS
- iJaNg Sarjan: nice to talk about this with u
- Trifon Trifonov: JMS is quite simplke to use.
- Trifon Trifonov: just send and receive messages.
- iJaNg Sarjan: yup I know
- iJaNg Sarjan: but if the connection between producer and consumer were lost
- iJaNg Sarjan: then the replication would be done right?
- iJaNg Sarjan:
- Trifon Trifonov: yes lost for the time when ther eis no connection but after that it will be restored
- Trifon Trifonov: that's why we have Local JMS Server and Remote JMS Server. they collect recieved messages and send only *when connection is online.
- Trifon Trifonov: Producer do not send directly to remote JMS Server.
- Trifon Trifonov: there are 3 layers.
- Trifon Trifonov: Local Producer --> to Local JMS -> Remote JMS Server -> Local Consumer
- Trifon Trifonov: connection between Local Producer and Local JMS server must be permanent.
- Trifon Trifonov: connection between Remote JMS Server and Remote Consumer must be permanenet too.
- Trifon Trifonov: but conection bettern Local JMS eerver and Remote JMS server can broke from time to time.
- iJaNg Sarjan: i.c
- iJaNg Sarjan: so the local JMS and Remote JMS could be some sort of recovery channel too right?
- iJaNg Sarjan: beside sending and receiveing msg
- Trifon Trifonov: yes right.
- iJaNg Sarjan: ok thanks trifon
- iJaNg Sarjan: got to go now
- iJaNg Sarjan: maybe we could have more discussion on this the other time
- iJaNg Sarjan: thans a lot
- Trifon Trifonov: sure.
- Trifon Trifonov: it was nice to chat with you.
- Trifon Trifonov: have a nice evening.
- I'learn a lot from him and have clear understanding on the JMS instead just read from wiki or other websites.
I also show him my idea and the suggest topology to him
http://red1.org/aaa/Diagram.doc - Topology that I suggested
