Analytic Information Server is a mobile, distributed, Lambda-oriented repository. It provides a mobile runtime environment and a mobile persistent repository database. "Intelligent Lambdas" infiltrate each location of a distributed application, personalizing and adapting the application to each user's needs. It is no longer necessary to send user data from the client interface back to the server for lengthy processing. Users interact with distributed "info-nimble" application interfaces, which perform in-depth analysis of the user's needs on-site, at the client. With its open and modular architecture, Analytic Information Server allows intelligent Lambdas to be easily integrated with Excel, Oracle, Java, as well as with existing in-house applications such as customer service, data warehousing, decision support, sales automation, and help desk.
Multiple concurrent users are supported as well as, transaction rollback with no arbitrary limitation on the size of a transaction, password protected data security with automatic record compression and encryption, data recovery, and a runtime dynamic database schema at each location.
Thousands of autonomous child Lambdas are created, each second, in an adaptive application. As child Lambdas are born, they are placed into an Lambda file cabinet. Each child Lambda has source code, compiled code, and knowledge information, which must be, stored somewhere. Often, there may be too many new Lambdas to keep in memory simultaneously. So, Analytic Information Server will move Lambdas from the file cabinet into memory and from memory back into the file cabinet during the course of data analysis.
The Analytic Information Server Repository has the ability to store Lambdas, Lambda opinions, Lambda knowledge, and Lambda data. Unlike many other database systems, Analytic Information Server supports no fixed database schema. Instead, the Lambdas implement the schema for their own data and for each server application. The client examines the interfaces for each Lambda to determine how to interact with the database. Analytic Information Server can be made to look like a relational database, like a text database, like an object database, or all of the above simultaneously (depending upon which Lambda is controlling the schema).
The Analytic Information Server repository supports the storing of mutating behavior in multiple repositories distributed across networks containing different computer architectures. Analytic Information Server Virtual Machine Lambda objects accomplish this feat, because the Analytic Information Server repository guarantees that a VM Lambda will operate identically on any computer where the Analytic Information Server engine resides. These stored behaviors are called Lambdas, and they are designed to invoke the analytic components in the Analytic Information Server function library to perform complex user defined analyses on the high volumes of data stored in the Analytic Information Server object repositories.
Fast Lambda oriented repository management is a corner stone of the Analytic Information Server database engine. The ability to store complex interrelated data structures with complex relationships fast, and to retrieve them fast, is essential to advanced, high volume data analysis. The Analytic Information Server object repository supports password protected data security, record compression and encryption, transaction rollback with no arbitrary limitation on the size of a transaction before rollback, automatic data recovery, multiple distributed volumes with a maximum of 2 gigabytes per volume and a maximum of 2 billion volumes supported. Each Analytic Information Server object repository supports automatic data recovery with the maximum possible data loss being limited to the object closures saved in the current transaction.
The Analytic Information Server general object repository is used in conjunction with commercial legacy database systems. In a typical ES-DSS application, large quantities of data are extracted from the legacy database and passed to the user defined Lambdas for analysis. These user defined Lambdas may need a FAST temporary archive (the Analytic Information Server General Object Repository) for portions of the extracted data. Furthermore, as the Lambdas develop hypotheses and increase their understanding of the extracted data, they may need a FAST persistent local archive for their internal knowledge representation. Often the internal knowledge representation of an Lambda is not in relational format; but, instead, is a complex network of analytic objects for which the Analytic Information Server General Object Repository provides an excellent persistence mechanism.
The basic unit of storage, in the Analytic Information Server General Object Repository, is the object closure (the original object saved, together with all the objects it reaches). Once saved in the repository, an Object Closure has a Repository Index which remains constant throughout the life of the object in the repository. Analytic Information Server places no limits on either the number of objects in an object closure nor on the complexity of their interrelationships. Literally any object closure of any size and complexity may be saved in a Analytic Information Server General Object Repository.
The ObjectRepository supports a simple associative memory database schema in which an object of any complexity is stored and retrieved using a key of any complexity. In addition to this native schema, Analytic Information Server supports programmable database schemas for each Object Repository. Programmable schemas are achieved by attaching a librarian Lambda to a repository. Once attached, the librarian has complete control of how the other intelligent Lambdas view the repository, and can develop a schema specifically tailored to the needs of the application. The librarian can easily implement an SQL relational style schema, a ODMG object style schema, a multivariate indexed style schema, or even a natural language database schema.
The Analytic Information Server Object Repository acts like a disk based associative memory in which data objects of arbitrary complexity are made persistent and associated with index objects also of arbitrary complexity. Storage is accomplished via the standard setq function, with the index object in the role of the key and the data object in the role of the new value to be saved. The setq function also supports various levels of object buffering, which are determined at creation time.
The Analytic Information Server Object Repository serves as a local scratchpad/blackboard area for multiple intelligent Lambdas during data analysis. Each Analytic Information Server Lambda is a Lisp script, which manages and coordinates a number of other tools and scripts in some specific type of analysis. The difference between an Lambda and a normal program, script, or procedure is fuzzy; however, the term Lambda is usually reserved for scripts which coordinate a specific type or style of analysis. An Lambda might train neural nets to recognize specific patterns in data, or an Lambda might apply a knowledgebase of rules against the data to prove a theorem or to form a concept about the data. In general Lambdas support a more intelligent interface than normal programs. Often they are capable of receiving and responding to messages, including requests for explanation, help, and even negotiations from other Lambdas in the knowledge system. Lambdas, during the course of their analysis, form complex opinions and models about the data. These opinions, often very complex networks of interrelated objects, represent the Lambda's viewpoint or vote as to the nature of the data. These Lambda opinions are stored in the Object Repository as a scratchpad/blackboard for other Lambdas, including meta or supervisory Lambdas, to read and digest.
The Object Repository supports transaction management with unlimited rollback before commit. Normally each set operation is an individual fully committed transaction; however, using the beginTransaction, commitTransaction, and abortTransaction functions, an Lambda can cause all set's back to the previous beginTransaction to be rolled back by using the abortTransaction functions instead of the commitTransaction functions.
The Object Repository supports transparent multiple concurrent Lambdas using the same database archive file, using the stop/go traffic light model. The Object Repository handles all exclusive locking without Lambda intervention; however, stop/go traffic light repository concurrency is designed for a small number of Lambdas occasionally sharing knowledge and information. There is no intent to handle the volume of concurrent retrieval and update transactions that a transaction database system would support.
The Object Repository supports transparent record level compression for all objects saved. Furthermore, each record may be encrypted with an optional numeric key at storage and retrieval time.
The basic unit of storage, in the Analytic Information Server General Object Repository, is the object closure (the original object saved, together with all the objects it reaches). Once saved in the repository, an Object Closure has a Repository Index, which remains constant throughout the life of the object in the repository.
There are two parts to the Object Repository: the first is an object archive (a database file on disk) and the second is the Repository Index. The Repository Index is an Analytic Information Server Directory object and saved on disk as part of the Object Repository. Each entry in the Repository Index is a binding consisting of a key and the location of the data on disk. Each record in the Object Repository is compressed using the LZW algorithm and then encrypted.
The Repository Index is simply a Directory, which is saved on the disk with the Object Repository. The keys in the Repository Index normally map to a value which is a pointer to the location in the disk file. Retrieving or saving a value to the Object Repository is key driven and requires two disk access:
The full range of object types, in the Analytic Information Server Data Model, may be saved in the Repository Index, and the full range of object types, in the Analytic Information Server Data Model, may be used as keys to index objects saved in a Analytic Information Server Object Repository. All Lambda scripting and modeling languages, available in Lambda Information Server, may be saved in the Object Repository. As an optimization feature, values may be optionally stored as immediate data in the object index using the saveImmediate function.
The Object Repository's data archive may contain up to 2 Gigabytes of objects and Lambdas. The Repository Index of an object repository must be entirely contained in memory. Each entry in the Repository Index requires approximately 80 bytes. A one-megabyte data index would support a Repository data archive containing approximately 12000 objects and Lambdas.
The Object Repository's Repository Index is a Directory. Each entry in the Repository Index is a binding of an object key and its value. The key can be any Analytical Object Of course, the Repository Index initially is very small and grows dynamically with the number of entries stored in the object repository, and can grow to any size limited only by available virtual memory.
The Object Repository offers persistent storage for the Analytic Information Server programmer. The full range of object types, in the Analytic Information Server Data Model, may be stored in the Object Repository, and the full range of object types, in the Analytic Information Server Data Model, may be used as keys to index objects saved in a Analytic Information Server Object Repository. All Lambda scripting and modeling languages, available in Lambda Information Server, may be saved in the Object Repository.
Analytic Information Server also supports programmable database schemas for each Object Repository. Programmable schemas are achieved by attaching a librarian Lambda to a repository. Once attached, the librarian has complete control of how the other intelligent Lambdas view the repository, and can develop a schema specifically tailored to the needs of the application. The librarian can easily implement an SQL relational style schema, a ODMG object style schema, a multivariate indexed style schema, or even a natural language database schema
Fast Lambda oriented repository management is a corner stone of the Analytic Information Server database engine. The ability to store complex interrelated data structures with complex relationships fast, and to retrieve them fast, is essential to advanced, high volume data analysis. The Analytic Information Server object repository supports password protected data security, record compression and encryption, transaction rollback with no arbitrary limitation on the size of a transaction before rollback, automatic data recovery, multiple distributed volumes with a maximum of 2 gigabytes per volume and a maximum of 2 billion volumes supported. Each Analytic Information Server object repository supports automatic data recovery with the maximum possible data loss being limited to the object closures saved in the current transaction.
The best pocket introduction to the essential elements of AIS Object Repositories is the following table of essential flow of control functional elements.
| (new ObjectRepository: fileName) | Repository creation/reopen statement |
|---|---|
| (set repository symbolicKey anyObject) | Indexed assignment statement |
| (set repository frame: frameID anyObject) | Direct frame assignment statement |
| (setf repository.symbolicKey anyObject) | Immediate indexed assignment statement |
| (ref repository symbolicKey) | Indexed access statement |
| (ref repository frame: frameID) | Direct frame access statement |
| (inspect repository) | Repository inspection statement |
| (beginTransaction repository) | Start an extended repository transaction |
| (checkPointTransaction repository) | Check point an extended repository transaction |
| (abortTransaction repository) | Abort an extended repository transaction |
| (commitTransaction repository) | Complete/accept an extended repository transaction |
| (attachLibrarian repository Lambda) | Attach an extended repository schema |
| (detachLibrarian repository) | Detach an extended repository schema |
Overview
The new function creates a new Object Repository, associates the Object Repository with a repository archive file, and allows for record encryption and other options. The styles of Object Repository creation are dependent upon the number and types of the arguments passed to the new function.
Type: Function
When To Use
The new function is used to create an Object Repository and allows the user to specify creation options.
Syntax: (new ObjectRepository: fileName)
(new ObjectRepository: fileName clear: key: code buffer: count)
Arguments
| ObjectRepository: | Mandatory argument specifying the type of object to create: ObjectRepository. |
|---|---|
| filename | The name of the database archive file to be associated with the ObjectRepository. If no such file exists, a new database archive file will be created. |
| clear: | (Optional) If the keyword clear: is present, the database archive file will be cleared immediately before any further processing. If no such file exists, a new database archive file will be created. |
| key: code | (Optional) If the keyword key: is present and it is followed by a number, the number is treated as an encryption code to use in record encrypting the database archive file. This same encryption key code must be used in all future references to the ObjectRepository. If the keyword key: is followed by the Boolean value true, the database archive file will be compressed when saved. If the Boolean value false is present, the archive file will not be compressed. The same compression key code must be used in all future references to the ObjectRepository. WARNING: If the ObjectRepository is later opened with a different key value from was specified during file creation, as a security measure, the ObjectRepository will be made unrecoverable |
| buffer: count | (Optional) If the key word buffer: is present, the numeric buffered object {count} must follow. The ObjectRepository will remember the last {count} objects retrieved to minimize disk access when required. As same objects are retrieved, the buffered object is returned and no disk access takes place. When a new object is retrieved, the oldest buffered object is thrown away to make room for the newly retrieved object. |
| Returns | The newly created ObjectRepository. |
Example 1
| (setq _path "") | ;;Reset the _path variable |
|---|---|
| (setq arcFile (new ObjectRepository: "myarchive.odb")) | ;; Set database archive file to "myarchive.odb" |
Example 2
| (setq _path "d:\\test\\") | ;; Set the _path variable to "d:\test" |
|---|---|
| (setq arcFile (new ObjectRepository: "myarchive.odb")) | ;; Set database archive file to "d:\test\myarchive.odb" |
Notes & Hints
[...under construction...]
Overview
The set function allows values to be stored in the specified ObjectRepository repository either by key or by direct frame id. Any value may be stored in the ObjectRepository and associated with a key. Both the key and the stored value may be of arbitrary complexity. An object may be removed from the ObjectRepository by setting the value #void in association with its previous key or frame id.
The set function also allows values to be stored in the specified ObjectRepository repository by direct frame id. Any value may be stored in the ObjectRepository and associated with a frame id. The stored value may be of arbitrary complexity. An object may be removed from the ObjectRepository by setting the value #void in association with its previous frame id.
Type: Special Form
When To Use
The set function is used to store objects into an Object Repository for later retrieval.
Syntax: (set repository key anyObject)
(setq frameID (set repository frame: frameID anyObject))
Arguments
| repository | Mandatory argument specifying the ObjectRepository into which the object is to be saved. |
|---|---|
| key | Mandatory argument specifying the key associated with the object is to be saved in the ObjectRepository. |
| anyObject | Mandatory argument specifying the object to be saved in the ObjectRepository. If the object to be saved is #void, then the specified key will be freed in the ObjectRepository. |
| Returns | The ObjectRepository after the save has taken place. |
| repository | Mandatory argument specifying the ObjectRepository into which the object is to be saved. |
|---|---|
| frame: | Mandatory keyword specifying that direct frame access is to be used with the object is to be saved in the ObjectRepository. |
| frameID | Mandatory argument specifying the frame id associated with the object is to be saved in the ObjectRepository. If the frameID is to small to hold the specified object, then a new frame id will be assigned. If the frameID is #void, then a new frame id will be assigned. |
| anyObject | Mandatory argument specifying the object to be saved in the ObjectRepository. If the object to be saved is #void, then the specified frame id will be freed in the ObjectRepository. |
| Returns | The frame id associated with the object saved in the repository. |
Example 1
Creating and initializing a new object repository.
(setq repo (new ObjectRepository: "test.db"))
(setq repo.Key1 "Hello World")
repo.Key1 Returns "Hello World"
(setq frameID (setq repo[frame: frameID] "Hello World"))
frameID Returns 1.0
repo[frame: frameID] Returns "Hello World"
(setq repo.Key1 #void)
repo.Key1 Returns #void
(setq frameID (setq repo[frame: frameID] #void))
repo[frame: frameID] Returns #void
Note: The set function saves object values into the object repository. These object values can be of any size and complexity.
Notes & Hints
[...under construction...]
Overview
The ref function allows values to be retreived from the specified ObjectRepository repository either by key or by direct frame id. Any previously stored value, associated with a key or a frame id, may be retrieved the ObjectRepository. Both the key and the stored value may be of arbitrary complexity. An object may be removed from the ObjectRepository by setting the value #void in association with its previous key or frame id.
The ref function also allows values to be retrieved from the specified ObjectRepository repository by direct frame id. Any previously stored value, associated with a frame id, may be retrieved the ObjectRepository. The retrieved value may be of arbitrary complexity. An object may be removed from the ObjectRepository by setting the value #void in association with its previous frame id.
Type: Special Form
When To Use
The ref function is used to retrieve previously saved objects from an Object Repository.
Syntax: (ref repository key)
(ref repository frame: frameID)
Arguments
| repository | Mandatory argument specifying the ObjectRepository from which the object is to be retrieved. |
|---|---|
| key | Mandatory argument specifying the key associated with which the object is to be retrieved from the ObjectRepository. |
| Returns | The previously saved object value retrieved from the ObjectRepository. |
| repository | Mandatory argument specifying the ObjectRepository from which the object is to be retrieved. |
|---|---|
| frame: | Mandatory keyword specifying that direct frame access is to be used with the object to be retrieved from the ObjectRepository. |
| frameID | Mandatory argument specifying the frame id associated with the object is to be retrieved from the ObjectRepository. |
| Returns | The previously saved object value retrieved from the ObjectRepository. |
Example 1
Creating and initializing a new object repository.
(setq repo (new ObjectRepository: "test.db"))
(setq repo.Key1 "Hello World")
repo.Key1 Returns "Hello World"
(setq frameID (setq repo[frame: frameID] "Hello World"))
frameID Returns 1.0
repo[frame: frameID] Returns "Hello World"
(setq repo.Key1 #void)
repo.Key1 Returns #void
(setq frameID (setq repo[frame: frameID] #void))
repo[frame: frameID] Returns #void
Note: The set function saves object values into the object repository. These object values can be of any size and complexity.
Notes & Hints
[...under construction...]
Overview
The global variable, _path, is reserved by the system to hold the directory and path name of the current working directory. The _path variable is set to the empty string "" at system startup. It is the programmer??s responsibility to initialize the _path variable. When the _path variable is referenced, the text contents of the _path variable are automatically attached to the beginning of database archive file named in the new function.
Type: Global Variable
When To Use
The _path is a global variable use to hold a path name string assigned by the programmer. If the _path variable is something other than a null string, the _path variable string will be prepended to the filename specified in the new function
Example 1
| (setq _path "") | ;;Reset the _path variable |
|---|---|
| (setq arcFile (new ObjectRepository: "myarchive.odb")) | ;; Set database archive file to "myarchive.odb" |
Example 2
| (setq _path "d:\\test\\") | ;; Set the _path variable to "d:\test" |
|---|---|
| (setq arcFile (new ObjectRepository: "myarchive.odb")) | ;; Set database archive file to "d:\test\myarchive.odb" |
Notes & Hints
[...under construction...]
Overview
The _saveTypes is a global variable that contains a Boolean value. The contents of the _saveTypes variable determines the calculation of the object closure when objects are saved to an Object Repository. At system startup, the _saveTypes variable is initialized to False. If the _saveTypes variable is later set to True, global value and type information are to be saved along with symbol objects.
Type: Global Variable
When To Use
Analytic Information Server Symbols have the two properties associated with them: type property and global value property. Typically, the preservation of type and global value information is needed when a Symbol is being used as a container. However, Symbols, are themselves data and may not be associated with a type or value. In the latter case, the type and global value properties are not important. The _saveTypes variable allows the programmer to control whether type and global information should be saved along with a Symbol object when the object is saved in an Object Repository.
Example 1
| (defineStructure x: Old1:) | ;; Global type info linked to x |
|---|---|
| (setq x #(1 2 3)) | ;; Global value stored in x |
| (setq y x:) | ;; y contains the Symbol x: |
| (setq db (new ObjectRepository: "test.db")) | |
| (setq _saveTypes false) | ;; Do NOT save Symbol global and type info |
| (setq db[0] y) | ;; Save record data only |
| (setq x #void) | ;; Destroy global value in x |
| (defineStructure x: New1: ) | ;; Destroy global type info in x /td> |
| (setq y db[0]) | ;; Read previously saved record into y |
| ((writeln "y = " y ", x = " x ", fieldsof (x) = " (fieldsOf x: ) ) |
The result output from the writeln statement is: y = x, x = #void, fieldsof (x) = #{New1: #void}
Note that the contents of the Symbol x: is #void.
Note that the type information of the Symbol x: is different from the original defineStructure, statement.
Example 2
| (defineStructure x: Old1:) | ;; Global type info linked to x |
|---|---|
| (setq x #(1 2 3)) | ;; Global value stored in x |
| (setq y x:) | ;; y contains the Symbol x: |
| (setq db (new ObjectRepository: "test.db")) | |
| (setq _saveTypes true) | ;; DO save Symbol global and type info |
| (setq db[0] y) | ;; Save Symbol global and type info |
| (setq x #void) | ;; Destroy global value in x |
| (defineStructure x: New1: ) | ;; Destroy global type info in x |
| (setq y db[0]) | ;; Read previously saved record into y |
| (writeln "y = " y ", x = " x ", fieldsOf (x) = " (fieldsOf x: ) ) |
The database archive file name is now y = x, x = #void, fieldsof (x) = #{New1: #void}
Note that the contents of the Symbol x: is the Vector #(1 2 3).
Note that the type information of the Symbol x: is the same as the original defineStructure statement.
Notes & Hints
[...under construction...]