Fig. 2

The default Smart Watcher Nifi processor group template. The Nifi processor group orchestrates a series of actions to manage data flow, extraction, and communication in a five-step process. Initially, a GenerateFlowFile processor, scheduled via Quartz-cron, creates an empty text file, which serves as a placeholder to initiate the data flow process and ensure that subsequent processors have a starting point. The empty text flow file is then managed by an ExecuteStreamCommand processor to execute a Python file that handles data queries, extractions, and transformations, storing results in an in-memory zip file to reduce server clutter, rather than storing the files on disk. Errors from the Python script or nonzero statuses trigger conditional routing by the RouteOnAttribute processor to a PutEmail processor, notifying the CogStack team via SMTP protocol for troubleshooting. Subsequently, an UpdateAttribute processor modifies the zip file name, and another RouteOnAttribute processor evaluates the zip file size to determine the routing path, ensuring relevant communication is sent to the requester and the CogStack team (who are only notified in the event when no data was found or an error has been encountered), either providing the extracted data or indicating no data was found, requiring further exploration