IFindIT Analytics

It is an ever-growing collection of services deployed in a private cloud that uses Big Data technologies to read, clean, transform, enrich, cross-reference, operate on, find, visualize, publish and deliver data of different types to generate value.
It comprises a set of integrated tools, predesigned templates and managed services that make it easier to create and manage information.
It also includes core products such as advanced data analytics and intelligent search.
It uses technologies such as machine learning, natural language processing, Spark, foundation models (generative artificial intelligence) and columnar databases, among others.
Given its level of maturity, it enables agile, efficient and cost-effective solutions that improve access to information, knowledge generation and informed decision-making, among other challenges

Broadly speaking, IFINDIT, as an analytics platform based on advanced search and designed for Big Data (it can process millions of records per hour), supports the following processes, among others:

Loading: Includes deployed, ready-to-use connectors. The agency must provide the data to be processed through a generalized view (fact table), to which IFINDIT will connect via a site-to-site VPN using one of these mechanisms:

• Connection to various types of database systems such as SQL, MySQL, Oracle, DB2, Postgres, Informix, Casandra, MariaDB, MongoDB and other databases through JDBC.
• Structured data in txt, Raw, xls, csv and parquet files.

Processing architecture: IFindIT is based on the Kappa architecture, because a data processing architecture designed to provide a scalable, fault-tolerant and flexible system is needed to process large amounts of data both in real time and in batch.
Kappa uses a single data processing system to handle both streaming and batch processing workloads; this enables more agile and simpler data processing while providing fast, reliable access to query results.

Columnar storage: This means that only the columns needed for a specific query are loaded, which improves the speed of queries that retrieve only a few columns and aggregations (e.g., how many people with which characteristics, over what time period, etc.).

Scalable distributed system: The platform can grow elastically. Given the high demand for information, it must support operations that may include loading data at a rate of millions of records per minute while serving queries that return results in a few seconds.

Massively parallel processing: The platform receives requests that may involve hundreds of millions of data points in a single query, so it must be able to process each query in parallel across the entire cluster.

Reports: Generates reports, aggregated data, columns, counts and SQL-style queries

Wizard-style query and report creation: Allows people without a high level of technical expertise to create reports, making it possible to split the workload and respond to requests quickly.

Data output:
The platform delivers data in the following formats:

• Non-aggregated data: xls, csv, txt, Json, XML
• Bulk data: data can be extracted in packages of millions of records with dozens of columns each
• Visualization: The platform can display data visually in dashboards, for both aggregated and non-aggregated data, on both desktop computers and mobile devices

Security: The platform provides role-based authentication and access control mechanisms. It integrates with Active Directory but also allows the creation of users local to the platform.

Robustness: IFindIT can be used by thousands of users simultaneously, responding to requests WITHOUT impacting the information systems that hold the data. Such impacts are especially evident in databases, where memory consumption is very high under concurrency.