NetPM

The problem it solves

Counters arrive from many OEM platforms in different formats (XML, CSV, ASN.1), on irregular schedules, at multiple granularities, with late and duplicate files. Making sense of that at scale, and trusting the result — is hard. NetPM standardises multi-structured counter data into a single explorable model, handles late and duplicate data inline without manual intervention, stores highly accurate granular data so performance drops can be tied back to configuration changes, and gives engineers counter- and KPI-level drill-down for root cause analysis. A distinguishing design principle: NetPM correlates on the fly rather than relying solely on pre-aggregated data, keeping groomed counters for long periods so KPIs can be recalculated as needed.

The ETL pipeline

Data moves through six stages.

1. Collect

   Continuous collection from RDBMS pull, pull/push file collection (3GPP XML, CSV), message-queue processing and web API requests, with swift onboarding of new sources.

2. Parse

   Multi-threaded, scalable inline parsing into a standardised format, with adaptable error handling and rapid parser development.

3. Load

   High-throughput inline loading into an analytics-optimised datastore; late and duplicate data processed inline; multiple granularity periods with rollup; file archiving and process tracking for audit.

4. Quality

   Rules-based validation and integrity checks, file and record audit control, and visibility of loading statistics — when data arrived versus when it was recorded.

5. Groom

   Counter-definition-based filtering to keep selected counters longer, an optimised partitioned schema, duplicate-record management and backfill of low-quality data.

6. Calculate

   Scheduled hourly and daily KPI calculation and aggregation for trend analysis and dashboards, with KPI recalculation and backfill.

Supported granularities: 1, 5, 15, 30 and 60-minute intervals, with automatic hourly, daily, weekly, monthly and yearly aggregation (SUM, AVG, MIN, MAX).

Data integrity you can see

NetPM builds integrity checking into the data itself. Files load as they arrive from the OSS, irregularly, and can be filtered at OSS and OSS-site level; after loading, consistency is checked against counter timestamps, and KPI counts over time show whether all nodes loaded for the period on screen. KPI numerator and denominator are stored separately per hour for each element and calculated after aggregation, which allows correct slice-and-dice on the front end. The Integrity Analyzer detects missing data in a KPI using cell counts, so files can be added and KPIs backfilled, and flags potential duplication.

A naïve average of the three percentages reports 3.4% — nearly 34× the true figure. That is why numerator and denominator are aggregated separately.

Building and managing KPIs

• Create and edit KPIs with full audit functionality and version control on each counter.
• Select counters by group drill-down or search; copy and rename groomed counters with user-friendly names.
• Add numerator and denominator formulas using multiple available functions; select node and element formulas from the raw data ID field.
• Pull a sample of raw data with filters and preview output before committing a KPI to the schedule.
• Attach dictionary metadata (latitude/longitude, state, region) for quick dashboard querying.
• Backfill fully supported — groomed counters from raw counters, KPIs from groomed counters, over a settable period.

Analysis & visualisation

• Raw Counter Analyzer — counter IDs as key/value pairs that can be sorted, grouped and filtered, in tables (Excel-exportable) or graphs, with built-in data-integrity indicators.
• KPI Analyzer — role-based and permission-controlled, with saved profile versions and filters; view and group multiple KPIs at once filtered on metadata such as region; polar graphs for month-on-month views; multiple panes sharing an x-axis with independent y-axis zoom; drill-down to cell level.
• Trending and capacity analysis — moving averages, comparison windows, front-end formula alteration, and a Trending Assistant that forecasts expected capacity from the low and high points of a KPI trend.
• GIS Analyser — UMTS propagation delay and quality, Voronoi mapping of a KPI or counter on a 2D cell layer, and transmission topology showing hub sites and their child nodes to identify areas affected by a hub failure.
• Worst Cell List — tracks how often a cell appears among the worst performers, and records the optimisation action, responsible engineer and region.

Group and local deployment

NetPM is designed for group operators. Each local operating company runs its own instance collecting from its OSS/EMS/NMS systems into a centralised per-OPCO database; aggregated Key Quality Indicators pass to a group server, giving the group a normalised cross-vendor view to compare operating companies.

Architecture

A backend ingests CM/PM and correlation data through the ETL pipeline, producing raw counters, groomed counters, and hourly and daily KPIs governed by rules for deduplication, archiving, data-quality handling, recalculation and aggregation. Data stores are ClickHouse and Postgres, with data flow built on Apache NiFi and bespoke components. The web front end covers visualisation, correlation, a dashboard builder, GIS and reports, plus admin tooling for ETL, KPIs, alarms, counters, rules and data dictionaries. Collection uses FTP/SFTP/RSYNC to a mediation server; integration to surrounding systems is via JDBC/API. Deployment on hardware or cloud.

Correlation & alarming

Where NetCM is also installed, configuration changes can be plotted against KPI changes over the same interval to investigate degradations and improvements — the core value of running configuration and performance on one platform. An Alarming Module triggers notifications, emails or script execution on range-based KPI thresholds. Access is protected by group-based profiles and authentication restricting screens, functionality and data per user or group.