Log Processing Architecture

Component Diagram

_images/725034d9f7699313a5befdd8c7c7929cc73c10e6974031a8472b8aaf0f4f80c1.svg

General flow of event information in the DataONE system. Events occur at Member and Coordinating Nodes and are aggregated by the Log Aggregation process that runs within the d1-processing application. Events are written to both a solr index and to JSON format log files. The log files are monitored by filebeat and forwarded to a logstash service. The logstash service applies geocoding, robots identification, and user agent matching before adding to an elasticseach instance.