External Information Model

The purpose of this proposal is to provide a new intermediate layer -- dubbed the 'External Information Model' (aka EIM) -- between objects and their external serialized representations. Decoupling the two in this manner means either can change format without affecting the other; a parcel developer doesn't need to know about external formats, as long as they write code that can convert items to and from EIM form. Whether we're dumping data to disk or sharing data across the wire, parcels will convert items to this new intermediate form which will then be converted to the appropriate external representation upon export. Having such a model also helps solve the schema evolution problem because it makes it easier for parcel developers to write upgrade code; they simply write code that translates a more-or-less fixed external format into their parcel's current schema. Because the external format is not used for normal application processing, it can be simpler and far more stable than the application-specific schema is likely to be.

When a parcel developer goes through the process of determining what attributes they want to share (or dump/reload), they figure out what "record types" would be required to store those attributes and then write methods which convert that parcel's items to and from those records. Think of a record type as a relational database table -- i.e. a ordered list of columns -- and records as rows. Next, for each record type the parcel writer defines a unique namespace name and registers the name with the sharing framework, providing two conversion callbacks -- 'inbound' and 'outbound' -- for that record type. When the time comes to import an item (whether by subscription, import, etc.) the external, serialized representation of the item is converted into to EIM form which consists of one or more record types, each with an associated namespace name. The sharing framework then steps through the record types' namespace names and calls the 'inbound' conversion callback that was registered on each. The callback is then responsible for creating/updating/deleting items.

Similarly, when it's time to export/publish an item, the set of namespaces appropriate to that item are determined, and the 'outbound' registered callbacks are called to convert the item to EIM form. Once there, the sharing framework hands the data to the appropriate conduit/format objects to serialize it.

Another new proposed concept is a "sharing schema" which consists of a name, a version, and a collection of namespaces that are mutually dependent. At the time a sharing conduit is being configured (for publish, subscribe, dump, reload, etc.) the appropriate sharing schema(s) are selected and noted in the conduit. We will ship with a sharing schema that covers all the out-of-the-box Kinds we define, and parcel writers can add their own.

Example 1:

A developer writes a Bookmark parcel, and defines Bookmark as a subkind of ContentItem, adding 'URL' and 'Last-Visited' attributes. Assuming that there is already a sharing schema defined which handles all the attributes of ContentItem, only a simple record type of three fields is required: UUID, URL, Last-Visited. The developer writes the two conversion methods:

def inbound(record):
   item = getItem(record[0])
   item.url, item.lastVisited = record[1:]

def outbound(item):
   return item.itsUUID, item.url, item.lastVisited

Next the developer chooses a namespace name, say 'http://example.org/schemas/bookmark/1.0', and registers the two conversion callbacks with it:

sharing.register('http://example.org/schemas/bookmark/1.0', inbound, outbound)

Example 2:

In the previous example we were able to add the new attributes "inline" in a single record type. However, a many-to-many relationship such as the "participants" relationship between CalendarEvents^? and Contacts requires a separate record type (like an RDBMS join table) with two fields: Event-UID and Contact-UID. If the list needs to be ordered, a third field could be added to maintain sort order. If instead the attribute holds a dictionary, then a third dictionary-key field could be added.

Example 3: Sample Records

Here are a few record type definitions:

class ItemRecord(sharing.Record):
    uuid = sharing.key(schema.UUID)
    title = sharing.field(sharing.TextType(size=256))
    triage_status = sharing.field(sharing.TextType(size=256))
    triage_status_changed = sharing.field(sharing.DecimalType(digits=11,
        decimal_places=2))
    last_modified_by = sharing.field(sharing.TextType(size=256)) # storing an email address
    created_on = sharing.field(sharing.DateType)

class NoteRecord(sharing.Record):
    uuid = sharing.key(schema.UUID)
    body = sharing.field(sharing.LobType())
    icaluid = sharing.field(sharing.TextType(size=256))

class TaskRecord(sharing.Record):
    uuid = sharing.key(schema.UUID)

class EventRecord(sharing.Record):
    uuid = sharing.key(schema.UUID)
    dtstart = sharing.field(sharing.DateType)
    dtend = sharing.field(sharing.DateType)
    location = sharing.field(sharing.TextType(size=256))
    rrule = sharing.field(sharing.TextType(size=1024))
    exrule = sharing.field(sharing.TextType(size=1024))
    rdate = sharing.field(sharing.DateType)
    exdate = sharing.field(sharing.DateType)
    recurrenceid = sharing.field(sharing.DateType)
    status = sharing.field(sharing.TextType(size=256))

...and here are some sample items:

Note (UUID 1)

• displayName = "Example note"
• body = "Example body"
• triageStatus = "done"
• triageStatusChanged = (float) -1164697997.0
• createdOn = 2006-07-13 12:26:00-07:00
• lastModifiedBy = UUID 2

Contact (UUID 2)

• createdOn = 2006-07-13 12:30:00-07:00
• contactName = UUID 3
• emailAddress = "morgen@example.com"

ContactName^? (UUID 3)

• createdOn = 2006-07-13 12:30:00-07:00
• firstName = "Morgen"
• lastName = "Sagen"

Example items in EIM form

2 tables:

Table 1: namespace http://schemas.osafoundation.org/pim/item

# (Item UUID, title, triage_status, triage_status_changed, lastModifiedBy UUID, createdOn)

• (1, "Example note", "done", -1164697997.00, "morgen@example.com", "2006-07-13 12:26:00-07:00")

Table 2: namespace http://schemas.osafoundation.org/pim/note

# (Item UUID, body, icaluid)

• (1, "Example body", "")