Proven Componenets¶
Proven consists of the following primary architectural elements:
- Exchange - Data collection, preparation, and distribution to Hybrid Store’s streaming environment.
- Hybrid Store – Message streaming and cache, with archival (TS, T3, Object). Messages are RDF sub-graphs. Stream processing support.
Inside Proven Member¶
Proven Exchange¶
- Data collection and preparation for distribution to Hybrid Store
- External - JSON and JSON-LD are currently the accepted formats
- Internal – JSON-LD (i.e. data represented as subgraphs internally)
- All disclosed content is verified for syntactic correctness. Each content type has an associated JSON-SCHEMA to perform this verification; a 2-part process:
- Proven-message is verified (i.e. wrapper; the proven part)
- Message content is verified. If content is JSON-LD then JSON-SCHEMA for JSON-LD is used to verify syntactic correctness.
- All disclosed content is transformed into a subgraph for semantic processing, as follows:
- Mime type is examined if JSON-LD then no further processing necessary
- For domain Knowledge content: A default JSON-LD context is provided in the outer object. This simply includes a @vocab setting using the message’s domain value.
- For “Proven specific” content: The pre-defined context for the message content type is injected.
- If the message’s outer object is an array, then the array is first encapsulated by an object before adding the context.
- All proven specific content types (including the wrapper) have an associated ontology and context definition.
- Message syntactic verification and semantic transforms are performed at point of entry and any issues are reported to Hybrid store’s Response stream.
- Exchange consists of 2 primary component types:
- Exchange Buffer: Have unique responsibilities in terms of disclosure item processing
- Module Exchange: Responsible for distributing a disclosed item to a “ready” ExchangeBuffer for processing (distribution can be at the module, member or cluster level)
- Following are the ExchangeBuffer types:
- DisclosureBuffer: disclosure item distribution
- ModuleServiceBuffer: services module requests.
- PipelineServiceBuffer: services pipeline requests.
- ResponseBuffer: distribution of response/results to domain resonse stream.
- ProvenanceBuffer: provenance generation and distribution
- RulesBuffer: rule-based inference and distribution
- Provenance capture is accomplished using the afore mentioned message ontologies and SHACL rules to generate PROV provenance assertions.
- Rules are also defined using SHACL and content specific ontologies.
- Each domain has its own provenance and stream.
- Each buffer is applicable to any domain; processing is determined by a message’s semantic description or Message Model (i.e. ontologies, context, rules, provenance, etc.)
- Disclsoure item paths are static and these paths are defined and used by a ModuleExchange.
- ModuleExchange’s are informed of the candidate ExchangeBuffers via module reporting making their lookup performant.
- Back pressure is to the caller.
- Exchange items that cannot be processed due to an “unavailable” ModuleExchange, are transferred to a Suspend stream to avoid data loss. These items are given highest priority once a ModuleExchange becomes available.