Log Processing Architecture

Component Diagram

General flow of event information in the DataONE system. Events occur at Member and Coordinating Nodes and are aggregated by the Log Aggregation process that runs within the d1-processing application. Events are written to both a solr index and to JSON format log files. The log files are monitored by filebeat and forwarded to a logstash service. The logstash service applies geocoding, robots identification, and user agent matching before adding to an elasticseach instance.