IDIOM Transaction Engine

IDIOM Transaction Engine

An Application to Manage Complex
Business Entities

Whitepaper February 2016

 +64 9 6308950 | @ [email protected] |  idiomsoftware.com
 2-2, 93 Dominion Road, Mount Eden, Auckland 1024

IDIOM Transaction Engine

Background................................................................................................................................................. 4
History........................................................................................................................................................ 4
IDIOM Published Articles ....................................................................................................................... 5
“Decisioning: a new approach to systems development”........................................................ 5
Requirements and the Beast of Complexity ................................................................................. 5
Decisioning – the Next Generation of Business Rules .................................................................. 6
The Role of SQL in Decision Centric Processing ............................................................................ 7
Taming the IT Beast with Decision Centric Processes .................................................................. 8
Where It Fits.............................................................................................................................................. 9

IDIOM Transaction Engine ...................................................................................................................... 10
Purpose................................................................................................................................................... 10
What is the Transaction Engine.......................................................................................................... 11
Transaction Engine Terminology........................................................................................................ 11
Component Overview ........................................................................................................................ 12
The Transaction Engine Core ............................................................................................................. 14
The Transaction Builder Platform........................................................................................................ 15
Generic Database ............................................................................................................................... 15
The Transaction Entity ...................................................................................................................... 16
Product Meta-Data.......................................................................................................................... 18
Permissioning and Process Control................................................................................................ 20
Batches............................................................................................................................................... 22
Activity Log ........................................................................................................................................ 23
Administration Engine.......................................................................................................................... 23
Public Access Engine........................................................................................................................... 24
Service Adapter Engine ...................................................................................................................... 24
IDIOM Product Engines........................................................................................................................ 24

How it Works .............................................................................................................................................. 24
Data ........................................................................................................................................................ 24
Events...................................................................................................................................................... 25

© Idiom Ltd 2015

User Requests..................................................................................................................................... 25
Machine Originated Requests ....................................................................................................... 26
BringUp Requests .............................................................................................................................. 26
Transaction Completion.................................................................................................................. 26
AutoSave............................................................................................................................................ 27
Processes................................................................................................................................................ 27
Workflow ............................................................................................................................................. 27
ITE Meta Data .................................................................................................................................... 28
Tasks..................................................................................................................................................... 30
Task Management ........................................................................................................................... 30
Bring Ups ............................................................................................................................................. 31
Putting It All Together............................................................................................................................... 31
The Generic Transaction ..................................................................................................................... 31
Versioning and Release ...................................................................................................................... 34
Transaction versions ......................................................................................................................... 34
Effective dating................................................................................................................................. 34
IDIOM Tools ................................................................................................................................................ 35
IDIOM Decision Manager ................................................................................................................... 36
IDIOM Forms........................................................................................................................................... 39
IDIOM Decision Tracker ....................................................................................................................... 40
IDIOM Mapper ...................................................................................................................................... 40
IDIOM Decision Manager Workbench ............................................................................................. 40
IDIOM Document Generator ............................................................................................................. 42
Conclusion ................................................................................................................................................. 43

Mark Norton | CEO and Founder | Idiom Limited

 +64 9 630 8950 | +64 21 434 669 | After hours +64 9 817 7165 | Australia free call 1800 049 004
 2-2, 93 Dominion Road, Mount Eden, Auckland 1024 | PO Box 60101, Titirangi, Auckland 0642, New Zealand
@ [email protected] |  idiomsoftware.com | Skype Mark.Norton

© Idiom Ltd 2015

IDIOM Transaction Engine

BACKGROUND

This Whitepaper introduces the IDIOM Transaction Engine. The IDIOM Transaction Engine is a
new IDIOM product that collates many new and existing IDIOM components into a common
enterprise scale platform that can be used to quickly and cost effectively build transactional
applications for a wide variety of complex business entities.

History

Over the past 15 years IDIOM has conceived, evolved, and demonstrated the effectiveness
of its ‘decision centric’ development approach, which leverages both decisioning and agile
approaches to dramatically simplify and strengthen commercial systems development. This
whitepaper describes the IDIOM Transaction Engine and how it supports the decision centric
approach.

Development of systems using the approach, and the IDIOM tools, has been repeatedly
shown to:

 simplify systems and their related development practices to reduce development time,
cost, and risk by substantial amounts

 increase business agility
 improve the alignment of business policy and computer systems
 improve runtime performance and reduce operational resource requirements.

The evolution of the approach has been chronicled in a series of published articles that
remain available as published.

For convenience, we have included a précis of the more important articles in the following
section. These précis’ provide insight into the evolution of the methodology and its rationale.
Links to the original articles are also provided for the interested reader. For further reading,
the most recent published paper describing the IDIOM approach and tools can be found on
our website as ‘Taming the IT Beast with IDIOM’, with an earlier version published on Modern
Analyst 1.

An outline of the IDIOM Transaction Engine is then offered; this is followed by a description of
each of the IDIOM tools.

The IDIOM products are derived from, and closely aligned with, the concepts described in
the articles, and together they provide a cost effective and risk averse path to successfully
building decision centric systems.

1 http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/3290/Taming-the-IT-Beast-with-IDIOM.aspx

© Idiom Ltd 2016 Page 4

IDIOM Published Articles

“Decisioning: a new approach to systems development”2

This seminal article (published in the Business Rules Journal, Jan 2007) provided the
background and conceptual building blocks for decisioning and decision centric
development. It highlighted the importance and benefits of decision analysis to good
systems design, and at the same time, noted its absence from the then current development
methodologies. It drew important distinctions between decisions and ‘business rules’, and
noted the relatively greater importance of decisions over processes as analysis subjects.

It laid the conceptual foundation for decisions (now defined as follows) to be a fundamental
unit of requirements specification.

Decision: A single definitive datum that is derived by applying business knowledge to relevant data for
the purpose of positively supporting or directing the activity of the business.

According to the article, these decisions are defined in the context of a decision model,
which is defined as:

Decision Model: An ordered assembly of decisions that creates new and proprietary information to
further the mission of the business.

And decisioning itself was defined as:

Decisioning: The systematic discovery, definition, deployment, and execution of automated decision
making.

Requirements and the Beast of Complexity3

[The ‘Requirements Article’]

This popular article (published in Modern Analyst, 2010, ~25,000 downloads as of November
2015) updated the above article, and targeted requirements specifications as a significant
weak link in the traditional systems development cycle. It presented a case for a new
approach to requirements gathering and specification that is based on analysis of business
policy to identify core business entities and their respective state changes, which are then
captured and described as decision models and ultimately implemented as transactions.
From the paper’s conclusion:

2 http://www.brcommunity.com/b326a.php Decisioning: A new approach to Systems Development

3 http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/1354/Requirements-and-the-Beast-of-
Complexity.aspx

© Idiom Ltd 2016 Page 5

“This ‘decision-centric development approach’ . . . focuses on a ‘missing link’ between business
strategy and operational systems – the Decision Model. The Decision Model is a new and important
requirements artefact that is accessible and understood by both business and development
practitioners, giving rise to a previously unobtainable level of shared understanding that can help
bridge the gap between business strategies and the systems that support them. The Decision Model
gives the business 'hands-on' control over the definition and implementation of its most critical IP – its
decision making know-how.”

Decisioning – the Next Generation of Business Rules4

[The ‘Decisioning Article’]

This article (published in both Modern Analyst and Business Rules Journal, 2013) noted that
Charles Forgy, the inventor of the rete algorithm (which powered a generation of business
rules engines), listed criteria that more or less excludes ‘transactions’ as an appropriate use-
case for the algorithm. Notwithstanding, rete aligned, constraints based rules engines
continue to be used in transactional systems, adding significant cost and complexity for little
value.

This article then expanded the scope of IDIOM’s business rules/decisioning concepts to
include the ability to dynamically create data that is aligned with the ‘idiom’5 of the
business; that is, with the nouns and noun clauses of the proprietary language (the idiom)
that we assert is always present when describing business rules. By definition, the idiom is
always proprietary to the author of the business rules because it is the need to define the
proprietary business rules that gives rise to the idiom.

Dynamic transformation from raw data to data that is compliant with the business idiom is
needed because a) it is rare that the real-world data is already in the required state for
adjudication by the rules, and b) the business idiom is fluid and ever changing – it is the
essence of business rules, changing whenever the business changes (c.f. the relatively static
nature of the real world data definitions).

The article also highlighted the separation of the doing and decisioning components within a
system:

“Service oriented architectures are inexorably trending towards separation of ‘doing’ components from
‘deciding’ components. The ‘doing’ components can be commoditized and outsourced. The truly
proprietary code, the code that captures business knowledge and IP, is being increasingly
concentrated in the ‘deciding’ components.”

4 http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/2713/Decisioning-the-next-generation-of-
Business-Rules.aspx
5 http://www.thefreedictionary.com/idiom [a. ‘A specialized vocabulary used by a group of people’]

© Idiom Ltd 2016 Page 6

The Role of SQL in Decision Centric Processing6

[The ‘SQL Article’]

The logical disconnect described in the above paper, between the original design intent of
the technology (rete in that case) and its subsequent use, is echoed with SQL. This article
(published in Modern Analyst (2014) and the Business Rules Journal (2015)) noted that SQL
was designed from the outset to address a specific class of problem – set processing.

This class of problem does not fit easily with transactions, and is to a large extent redundant if
the data is hierarchically structured around core business entities (and is stored as such). In
essence, the article made the observation that the ubiquitous SQL ‘join’ is fundamentally
flawed when used in a transactional context: Dependent tables should be joined onto their
parent entity; the parent entity should not be joined on to the dependent tables. Again, and
in apparent contrast with its original purpose, complex ‘join’ based SQL is routinely used to
underpin modern day transactions. And to close the circle on the business idiom described in
the Decisioning Article, we observed that the labels that are used to describe data in the
business idiom must describe that data in context; that is, the labels used for the data values
must include the relevant context qualifiers. If we are going to be able to implement business
rules that are described using the idiom, then it is necessary that we also describe the data in
terms that align with the idiom, so that the rules of data context are matched with the
vocabulary of the business idiom and vice versa. This is something not easily achieved using
SQL.

This correlation between the business idiom and data context is the essence of the linkage
between business policies and their implementation in computer systems. It is also the
backbone of business transactions, and in our experience, usually represents the major part
of a transaction’s internal work effort.

As a result, the SQL paper further extended the scope of IDIOM’s business rules/decisioning
concepts by highlighting data context as an integral part of a decisioning ‘requirements
specification’.

With a decision authoring tool that understands context, we can quickly and completely
define decision models that address the full scope of policy driven decision-making,
including validation, transformation, calculation, adjudication and workflow; and which
requires only ‘simple SQL’ (i.e. no joins) to provide complete and error free business
transactions of any size and complexity.

And these transactions will often execute faster than solutions developed using more
traditional approaches to use of SQL. This performance gain is primarily a reflection of the
way that SQL is used to support the transactions.

6 See, ‘The Role of SQL in Decision Centric Processing'
http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/3109/The-Role-of-SQL-in-Decision-Centric-
Processes.aspx’

© Idiom Ltd 2016 Page 7

Taming the IT Beast with Decision Centric Processes7

[The ‘Process Article’]

This article (published in Modern Analyst in 2015) builds on the earlier articles to address the
process perspective of the approach. It describes a process architecture and a matching
development approach for decision centric systems. It uses ‘business transactions’ as the
core application building blocks. A business transaction is defined as:

Business Transaction: The activity that is initiated by an event that changes the state of a business
entity, and which when complete leaves all systems compliant with all business policies relevant for
that entity in that state.

A business transaction embodies the complete life cycle of core business entities, and is
called and executed for each and every event that affects the entity.

The process complexity is reduced to a finite and known set of business policies in the
decision models, and a finite and known set of activities (usually implemented as services) in
the underlying application. Process orchestration is achieved by ‘just-in-time’, policy-derived
decision making to determine the ‘best-next’ activity or process. Each process or activity only
needs to calculate the ‘best-next’ step to ensure a complete and consistent overall process;
the only time best-next can be assessed is ‘right now’, using the current context and event
data to achieve immediate, real-time, process orchestration. We call this approach to
process management ‘binary BPM’.

The benefits are substantial. The architecture and its associated development approach
dramatically reduce the number of moving parts in the system, leading to an equally
dramatic reduction in the quantity of bespoke coding required. Throughout the article there
are observations regarding the simplification of systems development and execution that are
attributable to the approach.

When the approach is used to address complex business transaction processing, the
demonstrated results include substantial improvements in the key development metrics of
time, cost, and risk. These already encouraging results are accompanied by improvements in
business agility, better alignment between business strategy and operational systems, and
systems that are more transparent and more durable.

In the approach described by the article, business ‘Subject Matter Experts’ [SMEs] assume
hands-on responsibility for the definition, capture, and deployment of the business policies
that manage the entity life-cycles within the business transactions. By using the IDIOM tools to
implement the approach, these SMEs are empowered to take full and complete ownership
of the implementation of business policy within the operational business systems.

7 See, ‘Taming the IT Beast with Decision Centric Processes’
http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/3291/Taming-the-IT-Beast-with-Decision-Centric-
Processes.aspx

© Idiom Ltd 2016 Page 8

Where It Fits

It has become popular to refer to some systems as ‘systems of engagement’ and others as
‘systems of record’. Unfortunately these two classes of systems are sometimes the only classes
of systems considered, and so they become receptacles for corporate knowledge in the
form of rules and similar knowledge artefacts simply for want of any other place to put them.
In which case, how is this knowledge managed and shared?

Figure 1 – Classes of Systems

IDIOM’s view is that there are other equally ranked classes of systems as per the adjacent
diagram [Figure 1]. Most importantly, all classes of systems should defer to the ‘system of
applied knowledge, innovation, and learning’ that sits centre-stage. This architectural
pattern provides a formal place for corporate knowledge that is shareable through an ‘API

© Idiom Ltd 2016 Page 9

first’ approach. ‘API first’ is a concept that is expanding in popularity, and which is neatly
summed up by Logan, a guest commentator on the API Evangelist blog8 as follows:

“an API-first strategy is one where the API serves as both a pattern of an enterprises IT architecture as well as
catalyst for engagement. In a nutshell, an API-first strategy is a systems paradigm shift

- from a centralized, engineered mechanical system made up of subordinate parts serving as means to
deliberately designed ends

- to a decentralized, organic and evolving ecosystem where the individual actors can be spontaneously
enfranchised or co-opted by any other actor / set of actors to serve new purposes.

This allows the system to be more flexible, respond more quickly and behave more dynamically to suit
unforeseen purposes.”

API’s (especially in the form of Web Services) are a core feature of the Transaction Engine
design pattern, and are used to link the various classes of systems as shown in the above
diagram.

IDIOM TRANSACTION ENGINE

Purpose

The essential purpose of the Transaction Engine is to manage the full life cycle of complex
business entities, exclusively and totally. The range of applicable entities is extensive, and
includes at least the following.

 Information based products and services, for example loans, insurance policies, claims,
wealth accounts, pension accounts, etc.

 People and their interactions with the organisation, for example, customers, passengers,
patients, beneficiaries, employees (including payroll).

 Tangible entities; for example, property or physical assets that have conditional
maintenance and replacement cycles.

 Derived concepts; for example, resource and capacity demands.

By ‘full life cycle’, we mean all activities that affect the state of the entity, from acquisition
through disposition. The emphasis on ‘all’ is important – when all activities are managed
through the Transaction Engine, then the Transaction Engine can achieve its purpose
exclusively, so that no other system or system component needs to have any knowledge of
the internal structure or state of the entity. This exclusivity of knowledge and control allows
the Transaction Engine to be truly generic, to be configured by the business on demand as
and when business knowledge improves with regard to the management of its core business
entities. This is the heart of a learning organisation.

8 http://apievangelist.com/ 2016 Page 10

© Idiom Ltd

A generic, business configured application is also both agile and business aligned, by
definition, which satisfies two key objectives of the Transaction Engine concept.

And because the generic application components (i.e. prior to configuration) are vendor
supplied, we can reap the time, cost, and risk benefits of mass production; the benefits of
mass production and mass customisation together in a single ‘transaction engine’.

What is the Transaction Engine

The IDIOM Transaction Engine is a set of libraries that together form the basis of a generic
application that can manage the full life-cycle of any complex business entity. The
Transaction Engine is not an application per se – it always requires some degree of bespoke
development and configuration to become functional in a specific client context.

This is both deliberate and beneficial. Historically, vendor applications have struggled to
adapt to the many, many variations in client business and technology requirements that are
found in the real world. The consequences of this struggle range from large, cumbersome,
and expensive ‘omnibus systems’ (e.g. SAP); to systems that require significant customisation;
to a need for multiple systems; and/or the client simply accepts the constraints imposed by
the vendor application(s). Or, the client goes it alone and builds entirely bespoke. Each of
these options incur costs that can be orders of magnitude greater than the Transaction
Engine approach to provision of business applications.

The Transaction Engine addresses the problem of variable client needs in two ways:

 By using the IDIOM tools to address the client’s unique business context (‘what makes
you different’)

 And by completing the ‘last mile’ of application development only when the client
application and the infrastructure context is known.

In general, completing the ‘last mile’ of application development can be achieved in
parallel with the configuration of the business requirements using the IDIOM tools, so that any
plausible variation in the client’s technology landscape can be addressed without significant
time, cost, and risk at the same time as the client’s business context is being captured using
the IDIOM tools.

By the time the business requirements as described by the IDIOM tools are ready to be
deployed, the Transaction Engine is fully customised to the client’s specific needs.

It is IDIOM’s view that the variability of business content and technology landscapes
between individual clients has been under-estimated by the IT industry in general, and by
application vendors in particular. The outcome has been huge time, cost and risk for the
client, either in building bespoke applications, or in buying and then customising or
supplementing vendor applications. Or, the client simply lives in a world constrained by
vendor applications, lacking flexibility and any meaningful ability to differentiate.

Transaction Engine Terminology

We have chosen the term ‘transaction engine’ with some care.

© Idiom Ltd 2016 Page 11

We use the concept of an ‘engine’ widely in this document.

Engine: “An agent, instrument, or means of accomplishment”

from the Latin ingenium innate quality, talent, ingenious contrivance 9

An engine in this context is a separately identifiable and separately managed software
component that provides a ‘means of accomplishment’. However, it may not be complete
and ready for use in its delivered state: It may need natively coded customisation and/or
extension; and/or it may need the insertion of various IDIOM tool generated artefacts that will
define its ultimate business purpose.

The term transaction is used in a relatively orthodox manner to describe one or more
activities that collectively change the state of an entity (see Business Transaction definition in
the Process Article described above10). A state change is either made, or the entity is left
unchanged – there is no such thing as a partial transaction (by definition). The sum of all
state changes defines the life-cycle of the entity, completely and definitively.

Activity as described above refers to a secondary event that causes something to happen,
the details of which are unknown to the Transaction Engine Core. Each activity can be
thought of as a ‘black box’ from the perspective of the Transaction Engine; all required
‘black boxes’ can be orchestrated (on a ‘best, next’ basis) by the Transaction Engine without
the Transaction Engine understanding how those ‘black boxes’ perform their tasks.

Finally, for the sake of clarity, any reference to Subject Matter Experts [SME] in this document
refers exclusively to the business experts who are responsible for the configurations that are
captured by the IDIOM tools; that is, they are the users of the IDIOM tools.

Component Overview

The next diagram [Figure 2] provides an expanded view of the Transaction Engine in its role
as the dominant ‘system of knowledge, innovation, and learning’.

The yellow background indicates the Transaction Engine Core, which is a minimal framework
that provides similar services to that of an Enterprise Service Bus [ESB]; that is, it is a connector
and transport layer between the various end points. This basic capability is extended by a
number of capabilities that differentiate the IDIOM Transaction Engine – this includes
extensive use of decision models to make the transaction responsive to business policies, and
essential foundation capabilities including persistence, logging, and workflow management.

The Transaction Engine Core is implemented as a Windows Service that will start a new
transaction for any of four event types (shown as yellow arrows).

9 http://www.thefreedictionary.com/engine
10 Taming the IT Beast with Decision Centric Processes

© Idiom Ltd 2016 Page 12

Figure 2- The Transaction Engine as the dominant ‘system of knowledge, innovation, and learning’

© Idiom Ltd 2016 Page 13

The initiating events include:

 An external application can request a Transaction via Web Service Request.
 An authorised user can request a Transaction via the Administration Engine, or the Public

Access Engine.
 An earlier transaction execution can request a future dated transaction via the ‘BringUp’

table; the attendant polling service requests the future dated transaction on due date.

Each transaction runs in its own thread, with the number of threads being managed to
achieve the scalability objectives of the Transaction Engine.

The capital ‘I’ icon with a green border is present in engines that are dynamically configured
using IDIOM decision models.

There are three light green boxes with cogs shown in the diagram (right hand side), which
indicate separate and configurable secondary support ‘engines’. The two boxes that
include the IDIOM icon in the top left corner are engines that are distinct IDIOM products;
these engines are configured using their matching ‘builder’ applications in the Transaction
Builder Platform.

Where the support engines are visible to the open internet (Forms and services), we expect
them to be separated from the Transaction Engine Core by a Web Service, with decision
model controlled transformations limiting access to the core entity data.

The blue component, the Transaction Builder Platform, is the SME operated builder and
configuration platform for all of the IDIOM products. These products deploy to the
Transaction Engine by loading their respective configurations into the database.

The Administration Engine is a standalone web application that allows domain registered
users to manage and inspect the contents of the Transaction Engine. It can also raise events
on the Transaction Engine to update any selected Entity.

The Public Access Engine is a separate application for users who are not domain registered.
The access authority of these users is strictly limited by the application, and no crossover to
applications with greater authority is permitted.

The generic database completes the picture. The database is used directly by all of the
above components, and includes both user and system data.

The Transaction Engine Core

The yellow box.

An inbound event that is raised on the Transaction Engine Core is either for a new entity
instance, or a key is supplied for an existing entity instance; in either case, the entity record is
retrieved as a single XML document from the database (a pre-loaded pro-forma XML
document for a new instance) and loaded into memory along with its attendant decision
model(s). The event may supply event data, which is captured in a purpose built XML
Document (the ITE Meta Data, to be described later) and will be mapped onto the entity
record context by a decision model.

© Idiom Ltd 2016 Page 14

The decision models can also cause the Transaction Engine to raise secondary outbound
events on the supporting engines (right hand side above) by creating request Meta-Data
within each decision cycle; this Meta-Data is visible to the Transaction Engine in the ITE Meta-
Data (to be described later) and is inspected by the engine on each decision cycle.

The supporting engines include the IDIOM Forms and the IDIOM Document Generator, and
the Service Adapter Engine. In each case an attendant decision model will transform any
data that the supporting engine requires, both inbound and outbound.

The Transaction Builder Platform

The Transaction Builder Platform is the suite of IDIOM tools that can be used to configure the
Transaction Engine for a specific business purpose, for instance, an insurance product
platform, a claims platform, a loan platform, etc. Each of these tools is described later in this
document, and includes (in order, left to right):

 IDIOM Decision Manager
 IDIOM Forms Builder
 IDIOM Document Generator

The IDIOM Decision Manager Workbench is used to perform policy simulations and to
regression test policy changes prior to deploying new business policy. This tool is a standalone
application that is used by the SME’s to develop and quality assure business policy; it is not a
‘builder’ tool like the others, and so is shown separately.

A further capability of the Workbench is the ability to read, process, and (re)write a new
copy of the database on a large scale to mask data in the new copy. This can be useful if
the production copy is encrypted, and secondary copies are required for testing, trouble
shooting, or management reporting.

All of these tools can be used by appropriately trained SMEs or their respective analysts.

Generic Database

The Transaction Engine uses a generic database design. By this, we mean a database that
has no awareness in its design of what business entity data it contains, nor how that data is
structured.

This database design is key to allowing the business to be able to design entity data as and
when required, which is in turn an important element of the ‘business as content’ design
objective of the Transaction Engine itself.

The design requires that business entity data is stored as XML content within a simplified
database structure. For those familiar with data design, the adjacent database schema
provides an approximate outline of that database, and the following discussion offers some
insight into the underlying design pattern used for the Engine. This database design is
synthesized from several databases in large scale production use today.

© Idiom Ltd 2016 Page 15

Figure 3 – Proforma Database Design for Generic Transactions

We can now look at the various colour coded sections of the database in more detail.

The Transaction Entity

[Yellow Tables]

The focus of this database view is on the Transaction Entity, which is the centre element of
the yellow section.

In the following diagram [Figure 4], we can see the business entity data stored as ‘EntityXML’
in the Transaction Entity table. For commercial domains of even modest complexity, this
simple design concept can save us from having to build CRUD (Create, Read, Update,
Delete) programs for many tens of database tables; nor are we constrained by the
incredible development inertia that is implied by these tables. As a bonus, query
performance against this structure can be better than with standard ‘many table’ database
schemas because joins are rarely required. Also, some databases support ‘full text’ search on

© Idiom Ltd 2016 Page 16

the XML column, allowing Google like search capability over the full extent of the database
at little cost11.

Figure 4 – The Transaction Entity

A single auto-save record can be used to manage auto-save during any step in the
Transaction; or, if the user elects to save and exit without completing the transaction, then
the pending state of the Entity can be maintained in the AutoSave record. In this case, a

11 Our experience with SQL Server is that the performance cost of full text search is insignificant.

© Idiom Ltd 2016 Page 17

History record is not created, and the EntityXML master is not updated because the
Transaction did not complete.

The ‘Transaction Client’ is present to assist with access control. It is plausible that this record
may also need a ‘ClientXML’ column, however, no use cases at present require this. In all
cases to date, the EntityXML in the Transaction Entity record is a superset of all client data, so
that these records (client and entity) are one-to-one. This is expected to change for some
use cases in the future, hence the separation. (The ‘optional client parent’ is for an existing
use case where a complex entity is not actually a dependent of client.)

The ‘Related Entity’ table is to support the use case where a complex entity has a many to
many relationship with another complex entity, and these entities need to be collectively
involved in the transactions (i.e. the transaction spans entities).

Whenever a change to the EntityXML is confirmed on transaction termination, the
Transaction Engine will place a copy into the Entity History table, so that a full trace is
maintained of every state change for every entity, every time, forever. This trace includes
the latest copy of the EntityXML and the ITEMetaDataXML. The ITEMetaDataXML is the Meta-
Data that is used to control and record the processing of and by the IDIOM Transaction
Engine itself. Unlike the EntityXML, the structure of the ITE Meta-Data is known to the
Transaction Engine. The ITE Meta-Data has its own IDIOM defined schema that is described
more fully later in this section.

These XML records are compressed when inserted into the history table – these are not
normally searched.

Optimistic ‘soft locks’ are used to control concurrent updates. The nature of the entities that
are the target for the Transaction Engine means that update conflicts are highly unlikely.
Nonetheless, the possibility needs to be managed and this is achieved as follows. When the
Transaction Engine reads the EntityXML it will capture the current ChangeCounter. Database
locks are not used at this time because the transactions can be long running. In the fullness
of time, the Transaction Engine will want to write the record back – at this time, a database
lock is obtained and the change counter is re-read. If it is still the same value, then the
update continues; if it is different, the transaction is rolled back.

Product Meta-Data

[Green Tables]

The product Meta-Data is the data that describes the structure and logic of the transactions
that manage the state of the subject entities – that is, it is data that describes aspects of the
type of entity being managed, which we loosely refer to as ‘product’ (it may also be a
service or something similar), rather than data that describes an entity.

As we can see in the above diagram, the Transaction Entity is described by a Product
Version (see also Figure 5 below), which links the Transaction Entity with all of the Meta-Data
that is required to manage that entity’s life-cycle. Most such entities and life-cycles are
described as ‘products’ in business jargon, and we have adopted the term ‘product’ for this
purpose.

© Idiom Ltd 2016 Page 18

The product concept is used to describe and manage all of the product and service
offerings that can be supported by the Archetype. The Archetype is the root of the
Transaction Entity concept; it contains the context schema that describes all versions of the
EntityXML for all Transaction Entities that are associated with the Archetype (this schema
column is not used by any transaction based activity).

The Product Version is a direct parent of the Transaction Entity. It includes a pre-built blank
EntityXML document (as NewXML) for new instances of Transaction Entities; and an (optional)
configuration document. The Configuration Document is a document that holds various
parameters that will be used by the Product’s Control Model to make the decision
processing specific to this Product Version.

From the Product Version we can reach:
 The Product Control Model, which is responsible for managing all states for all entities that

are described by the Product.
 The Archetype, which contains the context data schema that describes all instances of

the EntityXML across all Products within the scope of the Archetype. The Archetype also
links to the decision model that dynamically provides a list of the valid menu options for
each Transaction Entity that exists within the scope of the Archetype; and the
Enumeration Model, which generates all pick-lists used by the Forms and Dialogs,
according to both context and version constraints.
 Forms that are built over the Archetype schema (the context data), with further links to
the models that support the Form. These include the Session Model, which controls the
Form’s Meta-Data.

The actual decision models referred to above are found in the ‘Binary’ column of the
Decision Model table [green], and are loaded directly from the database at runtime.

We include the Service Adapter table, so that we can identify the decision model that is
used to map data to/from the service. The Content Format attribute of this table is then used
to transcribe the decision model defined data into the required technical format (e.g. XML,
JSON, CSV, etc).

Finally, the Document Types are a list of the Document Types supported by the Document
Generator for this Archetype. The Document Generator is installed as a discrete application
(i.e. external to the Transaction Engine).

Figure 5 – Product Meta-Data (next page)

© Idiom Ltd 2016 Page 19

Permissioning and Process Control

[Red Tables]

The red sections of the model are used for workflow and process control. The Transaction
Engine includes its own security model for managing authentication (this is only provided
under NDA). It also provides a role based authorisation regime, which can be integrated with
an existing corporate Active Directory if required.

© Idiom Ltd 2016 Page 20

There are two major classes of users:
 Domain – Domain Authenticated Users [System Users];
 Public – Non Domain Internet Users [Transaction Clients].

Transaction clients are the ultimate beneficiaries of the system, and are the parent of the
Transaction Entities. These users may access the system using the Public Access Engine, which
is a separate application with limited ability to access the system. The system will always
recognise these users as ‘Public’, and will strictly limit their access authority to their own
transactions within the system, with further control at the decision model level (implemented
by the Archetype Menu Model) as required.

The Domain Users belong to the domain, and will be authenticated against their domain
credentials for each transaction that is attempted. These users will access via the
Administration Engine.

The Domain Users are further classified into Internal and External Domain Users. The Internal
Domain Users belong to the owner of the system (OwnerFlag = True on Party), and are not
limited in the scope of the Transactions that they can see. External Domain Users belong to a
Party other than the Owner of the system; for these users, data scope is restricted to the
Transaction Entities for which they have a current Related Party entry in the Related Party
table. Related Parties are generated by the Product Control Model, and are
validated/updated with every transaction.

Tasks can be generated by the Product Control Model. The Task Code links the Task to its Task
Type, which in turn links the Task to a Role and thereafter to authorised users. All users that are
associated with a Task are tracked via the Task Role table.

BringUps are future dated events that can be associated with different processes. The event
that the BringUp represents is invoked on due date by a polling service, at which time the
BringUp process will pass in any EventData from the BringUp record to initiate the transaction.
The normal BringUp will simply load the parent Transaction Entity’s EntityXML into memory
along with its associated Product Control Model (as defined by the entity’s ‘product
version’), thereby starting a standard transaction in the Transaction Engine.

Figure 6 – Permissioning and Process Control (next page)

© Idiom Ltd 2016 Page 21

Batches

[Pale Yellow Tables]

There are many uses for batch data, including:

 Financial records that are generated for the corporate ledger
 Bordereaux style records for third parties
 Management information and dashboard data for MIS use.

Batch Entries represent events that will be raised on external systems. Unlike the synchronous
updates that are captured in the ITE Meta-Data XML, the asynchronous batch entries must
be persisted in the Transaction Engine until the target system is ready to receive them.

At this time a Batch Header is created and all Entries that have not been transferred are
transferred and stamped with the Batch Header ID. During the transfer, the EntryDataXML will
be formatted as appropriate for the receiving system according to the Content Format
attribute of the Service Adapter table.

© Idiom Ltd 2016 Page 22

Note that reversal of a batch is easily achieved by updating the Batch Entries to remove the
link to the reversed Batch Header.

Figure 7 – Batches

Activity Log

Any process can generate an activity log for any entity instance (a row) in any table. If
required, before and after images of changed data can be inserted into the log record.

Administration Engine

The Administration Engine corresponds more closely with a traditional application. It has a
User Interface that allows domain authorised users (back-office, intermediaries as required)
to log on and access platform functionality. Within the Admin Engine, the Permissions Engine
is responsible for authenticating users, and then authorising them to the available functions.

These functions are either generic functions, or they are specific to the type of entity that the
Transaction Engine is managing. The former are managed in a traditional way, and this may
include linking to an existing enterprise Active Directory for the required role based
authorisation.

The entity-specific actions (for example, endorse an insurance policy) can only be made
available via the decision model(s) that control them (specifically, the Archetype Menu
Model), so that permissioning at this level becomes a decision model responsibility under the
direct control of the SME’s who are managing the entity life cycle; that is, the functions
currently available for any retrieved entity will be nominated by the decision model(s)
responsible for that entity.

The Search and Select Engine provides generic search facilities to locate required entities.
This can be by any (database) indexed column value(s), or by full text search. As noted, full
text search has been demonstrated to be both fast and light-weight in terms of resources

© Idiom Ltd 2016 Page 23

used. Scope restrictions are automatically applied by the Search Engine according to user
role

Public Access Engine

The Public Access Engine is for users who are not domain authorised. These users are
managed by a discrete application that supports a very limited range of functions.

Service Adapter Engine

The Service Adapter Engine is unlikely to be complete when delivered. Integration is
notoriously bespoke, and the Transaction Engine design accepts and adapts to this reality.
Therefore we expect to build web, queue and other bespoke service adapters as required to
meet the functional and integration requirements of the entity life-cycles for each business;
each new adapter that comes online can be used immediately as a new ‘activity’, and
invoked via SME authored decision models.

Any existing adapter can be reused at will. This reuse is facilitated by a subtle dislocation that
occurs within the Service Adapter. The distinct IDIOM decision model (the ServiceControl
Model) that is shown within the Service Adapter Engine is there to map the entity ‘context
data’ to the adapter’s specific schema. Therefore, provided that the context data and the
adapter data have an appropriate correlation, the SME designer can always read/write
data to/from the adapter. In a second and separate step, the adapter itself is then
responsible for mapping that data to any underlying technical implementation that the
correspondent end-point may require.

IDIOM Product Engines

The two IDIOM Product Engines shown in the diagram, managing forms and document
generation respectively, are further described in the later section on IDIOM Tools.

HOW IT WORKS

Data

The primary context data is the data that describes the entity en toto. The context data is
described using an XML schema (the archetype .xsd), which for a meaningful commercial
entity could easily scale to several hundred thousand nodes, with many layers of collection
depth. This entity specific data is NOT known to the Transaction Engine.

The ITE Meta-Data is data that is additional to and apart from the data that describes the
entity, and in this case is intended to be understood by the Engine. It is produced by the
decision models acting as proxies for the business SMEs that manage them, and is used to
control the Engine, and through it, all downstream activities.

On each execution cycle of the primary decision model(s), the Engine will inspect the Meta-
Data and act accordingly (this is analogous to content based routing in an ESB). For

© Idiom Ltd 2016 Page 24

instance, if the decision model determines that additional data is required from some
external service, the Meta-Data would be used to request this from the Transaction Engine,
which in turn will request it from the Service Adapter Engine. Similarly, on transaction
termination, which occurs at the conclusion of an entity state change, the decision models
can use this mechanism to ensure that all downstream activities are updated and aligned,
by indirectly posting updates to all affected systems.

Data that is generated by the models for consumption by these external services, or which is
provided by the services for consumption by the decision models, is appended to the Meta-
Data in the form required or created by the external service. This creates an obligation on
the decision models to manage the transformation and loading of this data, which also
implicitly validates it.

Events

There are three classes of events that will trigger a response from the Transaction Engine.
These are:

 User requests via the Admin Engine or the Public Access Engine
 Machine originated service requests
 BringUp requests from the BringUp polling service on the database

Each of these events causes a similar response in the IDIOM Transaction Engine: The context
entity is instantiated in memory, along with its decision model(s). Each event causes an
iteration of the rules, which will interpret the event and determine the appropriate response,
which will be signalled to the Engine via Meta-Data.

User Requests

All user requests are originated by the Administration Engine, which includes the Permissions
Engine and the Search & Select Engine, or the Public Access Engine.

The Administration Engine will present the user with a list of entities scoped by their
relationship to the entities. The database can be extended to include indexes on any
column(s) of the Transaction Entity table, which can then be used to find and order lists of
Transaction Entities (note in Figure 8 that the Transaction Entity column values are generated
by decision models and mapped automatically into the table). The data that is visible in the
lists comes from columns in the Transaction Entity table, so that the EntityXML does not need
to be opened for search and select purposes.

A range of generic role based activities can be accessed directly from the search list (eg
‘show history’, ‘open read only’, etc) without reference to the data inside the EntityXML.

However, some activities with update potential may need to be assessed by the Archetype’s
Menu Model, which will inspect the XML and offer menu options that are valid for ‘this user,
this entity, in this state’. To obtain access to these menu options, the EntityXML is instantiated
in memory.

© Idiom Ltd 2016 Page 25

If the user selects one of the options offered, then the event is raised on the Transaction
Engine, which will execute the decision model(s) appropriate for that entity.

At this time the decision model can request the Engine to open a direct communication
channel with the user via the IDIOM Forms Engine (an IDIOM Form). Communications can
flow between the decision model(s) to and from the user via the form.

Access via the Public Access Engine is similar, but simpler. There is no authorisation because
the only options present in the application are valid user options, and the data scope is
limited (by code) to the Transaction Entities that are dependent on the Transaction Client,
who in this case, is the user.

Machine Originated Requests

Other systems can raise events on the entity by raising an event via a web service, or posting
a message on a queue, or by some other bespoke means that is supported by the Service
Adapter engine.

The response is the same as described above: The Engine will load the entity, and its decision
models will handle the request. Because no user is present, IDIOM Forms are not able to be
requested.

BringUp Requests

The Engine includes a polling service which is regularly checking the ‘BringUp’ table in the
database for future dated events to raise – we assume that this is a scheduled daily task that
will occur at a system ‘quiet time’. When an event is raised, it is passed to the Engine along
with any context data that is held with the BringUp. Again, the Engine will load the entity, and
its decision models will handle the request.

As with all updates, the ChangeCounter is read with the EntityXML; if the ChangeCounter has
changed prior to updating the XML, the BringUp process is abandoned and will be retried.

Transaction Completion

The decision models will determine transaction termination when a state change is
determined (or rejected). The termination process is a discrete activity that runs under
commit control, which will:

 Re-read the Transaction Entity record.
 Check the Transaction Entity ChangeCounter to verify that no change has been

made since the transaction started.
 Update the Related Party table – relationships that have no longer present are end-

dated and can no longer be used to authorise access to the entity.
 Strip and post batch entries into their respective Batch Service.
 Merge new BringUps.
 Merge new Tasks.

© Idiom Ltd 2016 Page 26

 Update the generated column values for any table nominated by the
‘TransactionEntityColumns’ element in the ITE Meta Data, including the Transaction
Entity itself (this will include the updated ChangeCounter).

 Clear the auto-save, if any.
 Compress the XMLs and create a new Entity History record.
 Post a request for document generation of any required business documents (these

are posted back into the database).

AutoSave

AutoSave can be used to provide some protection from system failures during the use of
IDIOM Forms; and/or, it can be created by a specific save button in the Form. In either case,
the AutoSave record is sufficient to allow a transaction restart from the point of the last save.

All new transactions that open the Transaction Entity will first check the AutoSave as follows:

 Does an AutoSave exist?
 Is it for the same user?
 Is the Change Counter the same value as the Transaction Entity?

If yes to all of the above, then offer to open the AutoSave, otherwise open as a new
transaction.

Processes

Processes are handled in a non-standard manner in the Transaction Engine. For the reasons
described in the Process Article, pre-determined business processes are not a primary
requirements artefact in the IDIOM approach.

Processes are controlled by the ITE Meta Data and ‘emerge’ from the decision models that
are controlling this data. The traditional view of a process can be described as a pre-defined
and standardised sequence of component activities, which in the case of the Transaction
Engine may include Forms, Service Requests, Tasks, BringUps, and Document Requests. Each
activity may be required in response to multiple conditions in the underlying context; it is the
purpose of the workflow decision models to interpret this context data and to translate it into
the best-next activity in the sequence.

By sequencing these component activities using the decision models, any pre-defined
process can be made to occur, so that existing process documentation and the
implemented process will agree. However, many possible variations in the activity sequences
can evolve over time, so that the ‘source of (process) truth’ is actually to be found in the
decision models, in which case documenting the processes in arrears can also be attractive
– although this is likely to be manual and should be reserved for more important processes
that are likely to be subject to audit or other regulatory oversight.

Workflow

From Wikipedia:

© Idiom Ltd 2016 Page 27

A workflow consists of an orchestrated and repeatable pattern of business activity enabled by the systematic
organization of resources into processes that transform materials, provide services, or process information.[1] It
can be depicted as a sequence of operations, declared as work of a person or group,[2] an organization of staff, or
one or more simple or complex mechanisms.

From a more abstract or higher-level perspective, workflow may be considered a view or representation of real
work.[3] The flow being described may refer to a document, service or product that is being transferred from
one step to another.

Workflow has many forms. There are four main workflow threads within the Transaction
Engine, all managed by decision models.

Firstly, workflow within a state change activity is managed by the decision models within an
IDIOM Form, or if no form is present, then inside the decision models that are orchestrating
the state change. An IDIOM Form is an activity (aka a process) that can have complex
internal workflows; it can be of significant size and complexity, spanning many logical entities
and potentially thousands of data elements, involving many user interactions, and potentially
raising other secondary events.

Secondly, workflow across and between activities is managed by the Transaction Engine
acting in response to events raised in the ITE Meta Data. These events are created by the
attendant decision models, and cause the transaction engine to execute the activities (e.g.
one or more Forms, external services, etc.).

Thirdly, workflow that involves human operator intervention is managed by Tasks. Tasks are
events that are raised in the ITE Meta-Data to initiate human activities.

Fourthly, workflow that spans time is managed by BringUps. Bringups are future dated events
that cause the Transaction Engine to initiate any of the activities described above (although
Forms activities have to be initiated indirectly via Tasks or other outbound communications).

ITE Meta Data

Within the Transaction Engine Core the decision models are able to control all elements of
workflow that are relevant to the business entity through the ITE Meta-Data; the ITE Meta-
Data is always present for every transaction.

The Core will inspect the ITE Meta-Data on every rules cycle, and will respond to workflow
requests by initiating the relevant activities as required, including:

 Forms and generated Documents are requested using their respective elements in the
ITE Meta-Data schema (see next diagram).

 Real-time service based activities can be requested from the Engine by creating nodes
in the ITE Meta-Data XML that nominate the service required. The engine will then
execute the ServiceControlModel as defined by the Service Adapter (for the requested
service) to transform the context data into the data required by the Service. Any data
returned by the activity is added to the transaction’s available context data using the
same decision model to transform the inbound data.

© Idiom Ltd 2016 Page 28

 Batch activities are created as Batch Entries by creating nodes in the ITE Meta-Data
XML. The engine will then execute the ServiceControlModel as defined by the Service
Adapter to transform the context data into the required EntryDataXML.

 Future Dated Activities are controlled by generating Bring-Ups as ITE Meta-Data within
the primary decision cycle. The BringUps are stripped from the ITE Meta-Data XML when
the transaction terminates and loaded against the appropriate Transaction Entity in the
database. The Transaction Engine will launch a business transaction for each business
(transaction) entity that is nominated by a Bring-Up record.

 Manual activities are controlled by generating Tasks as ITE Meta-Data within the primary
decision cycle. The Tasks are stripped from the ITE Meta-Data XML when the transaction
terminates and loaded against the appropriate Task Type. The Task Type implicitly
assigns the Tasks to appropriate roles by the rules that create them. The Admin Engine
will automatically present the list of outstanding Tasks and prior assignments to users
whenever they are signed on.

Figure 8 – ITE Meta-Data XML schema

Each transaction’s primary decision cycle is responsible for orchestrating the workflow of its
subject entity in the final stages of each cycle, with the objective of aligning all systems now
and in the future with the current state of the entity. The ITE Meta-Data for every transaction is
recorded in the Entity History table.

© Idiom Ltd 2016 Page 29

Tasks

Tasks are created by decision models and placed in the ITE Meta-Data. When the EntityXML
is saved the Tasks are extracted and inserted as an ‘auto’ task into the database against the
Task Type. The Task Type ‘wires’ the Task into the Authorisation regime.

The insertion of the auto tasks is a merge function as follows:

 If the TaskCode already exists and the existing Task is open, ignore the new Task;
 If the TaskCode already exists and the existing Task is closed, then insert the new Task;
 If the TaskCode already exists, and the Task has not be assigned, then over-write it;
 If the TaskCode does not exist, then insert new Task.

For the sake of clarity, Tasks can also be created manually, although the Transaction Engine
offers no specific support for this process. The following discussion applies to all Tasks.

The Task columns are described as follows:

 TaskCode – Links to the Authorisation Regime;
 Source – auto or manual;
 TaskBringUpDate – the date that the Task becomes available for assignment;
 FollowUpDate – the date that a BringUp will be actioned for follow up activity;
 Status – (new), Under Action, Closed;
 Creation Date;
 Closed Date;
 Product and Entity Labels are for use in the Task display lists.

Task Management

Any System User (user) will have access to the Task lists, either at sign-on or via the menus.

The Tasks displayed in the list will be selected and ranked for all Roles for which the System
User Seniority is equal to or higher than the Task Type Seniority.

These Tasks can then be ordered by dates and status etc.

Depending on the system owner’s preferences, Tasks may be assigned or re-assigned for
action through self-selection (by the user), by others, or only by specific roles or levels of
seniority, or they may be operated on by any user on a first come, first serve basis.

By definition, an ‘auto’ task may cause some activity against the EntityXML. This activity will
automatically update the Task list in the ITE Meta-Data, as well as the BringUps.

Therefore, when a Task is created, it is common to also create a follow up BringUp; if the
BringUp is still in existence on the BringUpDate (also the Task FollowUpDate) then it will cause
the EntityXML to be opened and reviewed. This process may in turn generate a new and
more urgent Task (again with a follow up BringUp) or some other activity.

© Idiom Ltd 2016 Page 30

This interplay of Tasks and BringUps is how human operator workflows are integrated into the
automated workflows that are being managed by the product’s decision models.

Bring Ups

Like Tasks, Bring Ups are generated by the decision models into the ITE Meta Data and then
merged into the Bring Up table.

The merge process is simpler: delete any BringUps for this EntityID that have not been
processed, and insert all new BringUps.

PUTTING IT ALL TOGETHER

As noted in our preamble, the Transaction Engine is always incomplete when delivered.

Each instance of the Transaction Engine will require some degree of tailoring to marry it up to
the existing infrastructure. For the sake of clarity, this infrastructure can be zero for a new
business, or it can involve multiple legacy systems for a long established business.

Often, this retro-fitting process is limited to development of new adaptors in the Service
Adaptor Engine, and for existing corporate clients, linking the Permissions Engine into the
corporate authentication and authorisation infrastructure.

However, there is no inherent limit to the customisation, and this can include extensions to
the database and all that this implies.

While the Transaction Engine is being ‘fitted’ into the technical environment, the SME’s should
be busy configuring the business content using the IDIOM tools. This process is usually started
using IDIOM consultants in a ‘buddy’ style development mode, followed by a transition to
SME development without IDIOM assistance. The SME learning period is usually measured in
days to weeks.

The tailoring of the Transaction Engine, and the development and testing of the IDIOM tool
configurations, should occur more or less independently and in parallel to the technical
customisation. IDIOM recommends a maximum development window for both of these
activities of 3 months; beyond that, we suggest reducing scope. Our underlying principle is
that a modest but concise initial business scope is an important ingredient of success.

When the first iteration of the Transaction Engine has been deployed, ongoing adjustments
can be made by the SME’s acting alone, save perhaps for the comparatively infrequent
addition or update to Service Adaptors as may be required to maintain alignment with the
external technology landscape.

The Generic Transaction

The following diagram [Figure 8] shows a more complete illustrative transaction outline that is
capable of addressing many standard entity life-cycles, including those that we have used
as examples in this document (insurance policies, claims, loans, patient episodes, patient
clinical pathways, passengers, taxpayers, entitlements, et al).

© Idiom Ltd 2016 Page 31

The variations in the underlying entities, and by extension, their life-cycles, are all addressed
by the IDIOM tools within the ‘Entity Builder Platform’ (now referred to as the ‘Transaction
Builder Platform’). The SME has extensive control over the definition of these entities, and of
the actions that form their respective life-cycles. Provided that the various service end-points
are available to supply data to the transaction on request, and to respond to the updates
generated by the transaction on termination, then the transaction itself can be a black box.
For instance, a type ahead address lookup to verify property details could equally be serving
an insurance policy, a claim, or a loan; it is immaterial to the service which of these it is. The
majority of the development effort in building the transaction is in the decision models, which
are compiled and executed behind the standard interface that is built into the generic
business transaction itself. Independently measured projects have assessed this declarative
development effort as being at least 20 times more efficient than traditional SDLC
approaches to achieve the same end.

The generic IDIOM Transaction Engine uses the Product context to look up and load the
relevant decision models based on the nature of the entity that it is managing and as the
specific event and context requires.

The generic solution could also use the IDIOM Forms engine to run web forms to interact with
the users of the system (customers or back-office). And the Document Generator can be
deployed as a service within the host environment to generate and serve business
documents.

Note that IDIOM Forms and Document Generator Engines use IDIOM decision models to
drive their respective runtime Meta-Data, which is what makes these runtime engines truly
generic. The ongoing development of the decision models, forms, and documents requires
the builder versions of each of these tools. These are shown inside the Transaction Builder
Platform. The Workbench is also available in this platform to assist with policy modelling on
the one hand, and regression testing on the other.

Figure 9 The Transaction Engine at work (next page)

© Idiom Ltd 2016 Page 32

© Idiom Ltd 2016 Page 33

Versioning and Release

Continuous, business led development

An important objective of a transaction/decision centric application is continuous
development and release of business policy by business owners, or more specifically, the
SMEs. This business policy is usually focused on management of complex entity life-cycles.

Policy adjustments should be able to be developed, tested and deployed at any time. The
intent is not to actually deploy changes (say) daily, but to be able to deploy new policy at
any time the business requires, without the delay and risk inherent in traditional ‘code-drops’.
All parts of the business transaction, including all policy changes, are effective dated and
cumulative forever. The business transaction is always able to operate ‘as at’ any prior date.
The IDIOM approach also allows effective dating that spans changes in the entity schemas.

Continuous development and deployment is helped by IDIOM’s zero script, fully declarative
approach, coupled with a strong focus on versioning of and within all of the IDIOM
components, so that continuous, rapid change can be sustained – perpetually.

With a transaction/decision centric application there is a new dichotomy in versioning: the
schemas, forms, rules, and business documents, and all of their associated reference data,
can all be deployed externally to, and independently of, the code-base of the Transaction
Engine.

The continuous versioning that is inherent in the IDIOM approach is designed to support
nimble, continuous, perpetual, change in a decision centric application.

Transaction versions

A ‘transaction version’ is the net effect of all of the rules that implement the entirety of the
business policy as it was defined at the point in time that is defined by the transaction’s
effective date. Transaction versioning is therefore implicitly an effective dating regime.

Each change in the rules or of any other aspect of the declarative design within a business
transaction implies a version update for that transaction, and by extension, for the product or
service that it supports. All such versions always remain valid and operational, subject to the
effective date of the transaction at runtime.

Business transactions can also be versioned on criteria other than effective date through
configuration or rules changes – for instance, the term for specific conditions might be
negotiated so that each customer might see a different version of the same transaction rules
on the same date. This is achieved by dynamically setting the effective date using rules that
apply the specific conditions.

Effective dating

Effective dating is fully supported by the IDIOM Decision Manager.

An effective date is always present during the execution of decision models, which may be
set by default (i.e. today); or by the calling application; or by rules within the model itself (it is

© Idiom Ltd 2016 Page 34

possible to change the effective date during the processing of a decision model using
business rules). The effective date that is set by/for the transaction as above is used to
automatically select every aspect of the decision model at runtime. All aspects of the
decision model, from the complete model itself down to single reference-data values and/or
rules, are start and end dated as required.

Effective dating is not supported by the W3C XML Schema standard, and since XML
Schemas are the basis of IDIOM forms, this also limits effective dating in forms. However,
effective dating in the schemas and forms can be emulated by the rules that process them.
For instance, effective dating the presence of an element in a form is achieved by effective
dating the decisions that instantiate that element.

Finally, enumerations (i.e. pick-lists etc.) pose a specific versioning problem in forms. While the
Schema standard does support enumerations, these cannot be used because of context
and effective dating constraints which are not supported. Therefore, enumerations in an
IDIOM Form are typically generated by purpose built decision models (the Archetype’s
‘Enumeration Model’), so that the enumerations used in the forms are selected at runtime
according to both context and effective date.

As we have noted, the business transaction concept is generic. When implemented as
described, this means that entirely new business entities and their related Meta-Data can be
introduced at any time without changing the host application code. For the sake of clarity,
new types of entities, including new product types, can be introduced and, unless new
activities are required, can be immediately fully functional within the Transaction Engine.

IDIOM TOOLS

As you might expect, IDIOM has been fine-tuning its flagship IDIOM Decision Manager
product since its launch in 2001 to fully support the concepts outlined in our series of articles.

The ‘idiom’ is the proprietary language that is used to describe the essence of any business –
it is used to describe its business policies, its decision making, and its business rules. And in so
doing, the idiom refers to the data in the context required by the rules. In fact, the name of
our company was an early reflection of the importance of the business ‘idiom’, and its
relationship to both business rules and data.

The challenge lies in mirroring this context within the various executables that must implement
it.

Defining, capturing, validating, & automating the business ‘idiom’ is the essence of our business and
of our flagship product IDIOM Decision Manager.

The IDIOM Decision Manager Workbench is an industrial scale, ready-made batch
application processor for large scale, high performance batch processing of automated
decisions.

© Idiom Ltd 2016 Page 35

A transaction often requires dialog with a human actor. A dialog is easily managed by
building an IDIOM Form over the business transaction’s entity and event data – the same
data that is available to the decision models that interpret and maintain it.

An IDIOM form includes the ability to execute IDIOM decision models inside the form on a
large scale, field by field if necessary, to ensure a fully reflexive user experience.

Finally, the transaction end state might need to be reported via one or more business
documents. The IDIOM Document Generator can be used to generate complex business
documents under the control of decision models, using only Word authored text and images
as additional design input.

IDIOM Decision Manager

 IDIOM Decision Manager is a tool for graphically modelling, testing, and deploying
business decision-making logic – without programming!

 A tool for the policy maker, not the programmer.
 IDIOM automates complex decision-making at the enterprise level, deployable as

industrial strength stand-alone components.
 In day-to-day practice it is used by SMEs, or analysts working closely with them.
 Together they model the business policies in terms of both data and decisions (see

Decision Model below) before moving on to define the underlying logic that binds them
together (see Formula below).
 The decision logic and data are usually modelled together in a single combined process
of analysis and definition.
 The data model and the decision model share the same ‘context awareness’, with
current-context and context boundaries visually highlighted at all times within the
development palettes.
 Testing of the models is available at all times within the development palettes
themselves; full regression testing (incl. ‘model answer’ differencing) is available in the
on-board ‘Test Executive’ (not shown).
 Deployment as 100% generated software components is fully automated and ‘without
fingerprints’.
 Example below is a small but real model drawn from a city council implementation of
policy that calculates financial contributions to be paid by property developers.
 The problem domain is decomposed using a ‘mind mapping’ approach until we reach
the atomic units that we call decisions (rounded boxes).
 This ‘decision model’ and the adjacent data model (left hand panel below) are
demonstrably aligned and integrated through shared context – validating and
strengthening both.
 The data model defines the entity at rest; the decision model defines the valid state
transitions. Together they completely define the entity life-cycle and required business
policy.
 The atomic ‘decisions’ provide an easy entry point for specification of the underlying
rules details via the Formulas (see next).

© Idiom Ltd 2016 Page 36

 The underlying rules details are easily captured using a ‘Lego like’ drag-and-drop
approach that is ‘more fun than playing golf’ according to the CEO of one of our largest
customers – there is no scripting or coding required to build formulas.

 The rules can be tested immediately within the IDIOM Decision Manager palettes.
 When finished, IDIOM Decision Manager generates computer source code (C# or Java)

with a single button click, to be called by any application at run-time using any of a
wide variety of supplied interfaces and wrappers (in-line, dll, web service, queue service,
and many more).
 And at the same time, it generates the models into business readable PDF
documentation.

Key points of difference

 IDIOM’s decision models do for decisions what data models do for data – a powerful
abstraction that makes the underlying complexity visible and manageable.

 The models allow data transformations and more traditional business rules to be
intermingled. Business rules acting alone are severely limited in their ability to fully resolve
complex commercial problems – invariably, in-line data transformations are necessary to
facilitate the calculate/adjudicate/act behaviour of business rules.

 Context is continuously displayed and actively managed.
 Decision models that incorporate both data and rules behaviour enable a further critical

capability that is unique to IDIOM Decision Manager – the models can be fully tested
using real-world cases directly in the builder palettes. No external technology or
application support is required to empirically prove the completeness, consistency, and
correctness of the models.

Key points of innovation

 Fundamental redesign of the traditional SDLC by fully separating the development and
automation of business policy (deciding) from development of the system’s activities
that support it (doing).

 Use of IDIOM is effective in spawning a ‘Business Policy Development Life Cycle’ that is
managed independently of and alongside the traditional Systems Development Life
Cycle.

Key value propositions

 100% alignment of systems based decision making with business policy, because the
business owners have hands-on custody and control of the policy definitions actually
used by the system.

 Increased agility with reduced business risk through business modelling and empirical
testing of policy definitions prior to automated generation and implementation.

 Significant reduction in the business cost of developing and implementing automated
business policy.

 Further reduction in software development cost, time, & risk through reduced system
complexity, fewer moving parts, & clear separation of concerns.

© Idiom Ltd 2016 Page 37

Figure 10 IDIOM Decision Manager - Decision palette showing a Decision Model and Schema

Further benefits

 Full auditability of policy changes, and visibility of policy implementation through the
graphical models and logical English PDF generation.

 Decision model artefacts can be traced to/from ‘sources of truth’ in underlying business
policy documents (Word, Excel) using the IDIOM Tracker for requirements traceability.

 The IDIOM Decision Manager Workbench is available to experiment with and further
develop policy independently of the underlying systems and of the SDLC.

 Automated, robust, industrial strength deployment on any scale that can be supported
by the ‘host application’ and its underlying platform.

 Simple injection into legacy systems leading to eventual legacy replacement

© Idiom Ltd 2016 Page 38

Figure 11 IDIOM Decision Manager - showing the Formula Palette

IDIOM Forms

IDIOM Forms is a tool for generating web2.0 forms that embed the power of IDIOM decision
models into the form itself. As the user navigates through an IDIOM Form, the integrated
decision models can be executed on an element by element basis to make various business
calculations and assessments; and to modify the form’s Meta-Data. Control of the form’s
Meta-Data allows the decision models to dynamically adjust the visible shape and content
of the form on an element by element basis.

A single IDIOM Form can be used to process complex business transactions (e.g. an
insurance application, a clinical pathway, a loan application) through to closure, including
such functions as validation, transaction acceptance, costing or pricing, up-selling, and
determination of subsequent workflow amongst others. Usually, a single form is used to
manage the full life-cycle of the subject entity (i.e. not just individual state changes). The
IDIOM Forms Engine (the runtime component of IDIOM Forms) is production hardened after
many years use on thousands of servers throughout the NZ health sector. The deployed
IDIOM Forms Engine became the basis for the NZ Health Information Standards Organization
(HISO) forms standard and is currently the most compliant under that standard. The Forms
Engine is now also widely used in the finance sector for insurance underwriting and claims
management.

© Idiom Ltd 2016 Page 39

IDIOM Decision Tracker

The IDIOM Decision Tracker is a tool to map Microsoft Word and Excel policy documents to
IDIOM decision models for full bi-directional traceability between corporate policy definitions
in the Microsoft documents and their actual implementation as IDIOM generated decision
engines.

IDIOM Mapper

The IDIOM Mapper builds an in memory XML document that mirrors the database structure
that it is reading from. It generates ‘simple SQL’ from an XML configuration document, and
then executes a full round trip transaction cycle: from database to XML; then rules execution
(multiple decision models); and from XML back to the database if required. All done
extremely quickly, either in single transactions or in batch. The Mapper process is thread-safe
and can execute in many process streams for scalability that is limited only by the database
and its underlying platform. The Mapper is not normally used within the Transaction Engine,
but may be useful for tight legacy systems integration.

IDIOM Decision Manager Workbench

The IDIOM Decision Manager Workbench is a business operable application for running
decision models across enterprise databases on a large scale without the need for IT
technical support.

 A generic batch processing platform for use by IT and/or business operations.
 A platform for the auditor, the business policy manager, the product manager, the

corporate business strategy manager and the IT manager.
 An audit tool for identifying, reporting, and managing anomalies, errors, and issues of

concern in any database.
 A simulation tool for comparing the performance of current and ‘to be’ versions of any

automated policy.
 A tool that can intelligently modify the inbound data to simulate changes in the make-

up of inbound transactions over time.
 A data masking tool, able to replicate large scale databases with all personally

identifying details masked from its own library of several million fake identities.
Relationships between identities in the source data can be maintained, notwithstanding
the use of fake identities.
 A system conversion tool: able to read data from one system in its proprietary format;
intelligently transform it through one or more decision models; and then output it to a
new system also in its proprietary format. Millions of entities described by hundreds of
tables can be supported.

© Idiom Ltd 2016 Page 40

Figure 12 IDIOM Decision Manager Workbench

Examples of use

 Superannuation Fund Administrator: Perform all period end processing, including fee’s,
insurances, member adjustments, entitlements, and reporting, for several hundred
thousand fund members.

 Insurer: Compare the current underwriting policy with a proposed underwriting policy
across a recent portfolio of 500,000 insurance policies to determine potential changes in
the rate of referral and its attendant costs.

 Superannuation Fund Administrator: Run 100’s of distinct audit tests across 1million
member accounts on a daily basis to independently verify the production data.

 City Council: Compare this year’s rating policy with next year’s proposed rating policy
for each property in a city of more than 500,000 ratepayers to identify outlier changes in
the rates actually charged.

 Insurer: Dynamically modify key attributes of real transactions to simulate changes in the
make–up of in-bound business, and assess the impact of these changes across an entire
portfolio.

Enterprise class features

 Full authorization and audit controls down to the field level for all users of the platform.
 Seamless operation across multiple user definable environments, for instance

Development, UAT, Simulation, Production.
 High performance, including parallel processing on a large scale across multiple

machines, for instance running up to 24 Processes on an i7 class CPU, with multiple CPU’s
possible, each logging back to the Workbench for centralized management.

Measured performance

© Idiom Ltd 2016 Page 41

 Daily pass of 1million pension fund members each comprised of a join of 20+ complex
member related tables (~one billion rows in total).

 10’s of decision models implementing hundreds of individual member tests.
 Alerts captured and reported daily for start of business.
 Run daily in 2 hours using one i7 class processor, with data drawn from an IBM iSeries.

IDIOM Document Generator

The IDIOM Document Generator is a tool to collect and collate Word documents (which are
used as document ‘templates’) and many associated Word text fragments [‘blurbs’], and to
assemble these into complete documents under the control of a decision model.

At document design time, the Document Generator inspects the templates and blurbs and
builds a list of all blurbs (including nested blurbs) that could be used for each candidate
document; and it also builds a list of all of the variables declared within them. It then places
these two lists (blurbs and variables) into an XML ‘control document’.

A decision model is then built that will extend this XML control document at runtime when the
specific business entity context is known. This decision model will analyse the business entity
context data in order to set flags to control the insertion (or not) of each blurb; and it will
derive the substitution values for each of the listed variables.

At runtime, when the actual business entity data is provided and the decision model has
executed, the Document Generator will read the tags in the control document to insert the
desired blurbs into the template, followed by substitution of all variables, so that large and
complex documents can be generated automatically under the control of the decision
model to reflect the exact circumstances of the underlying business entity. The Document
Generator also allows for insertion of images and for calling bespoke programs to insert
additional generated text or images. Documents can be generated into a number of
formats, including Word and PDF. In day to day use by SME’s, the Document Generator only
requires standard Microsoft Word and IDIOM Decision Manager authoring skills to define new
documents.

© Idiom Ltd 2016 Page 42

Figure 13 IDIOM Document Generator

CONCLUSION

This whitepaper describes the rationale, structure, and use of the IDIOM Transaction Engine.

For prospective users of this innovative platform, all that is required is to undertake an analysis
to determine the scope of bespoke tailoring required, and of the IDIOM tool configurations
that are required to instantiate your business within the engine.

IDIOM will be pleased to work with you to complete these two requirements, and to present
them in a separate schedule. At that point, we will be ready to start development, and you
should be operational by the end of the next quarter – by definition.

Mark Norton | CEO and Founder | Idiom Limited

 +64 9 630 8950 | +64 21 434 669 | After hours +64 9 817 7165 | Australia free call 1800 049 004
 2-2, 93 Dominion Road, Mount Eden, Auckland 1024 | PO Box 60101, Titirangi, Auckland 0642, New Zealand
@ [email protected] |  idiomsoftware.com | Skype Mark.Norton

© Idiom Ltd 2016 Page 43