Skip to content

Commit 8173067

Browse files
Add opt-in STTP startup diagnostics and benchmarks
1 parent 1d76f6f commit 8173067

12 files changed

Lines changed: 933 additions & 0 deletions

File tree

‎docs/MetadataScaling.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Metadata scaling and 32-bit indexes
2+
3+
## Local reproduction
4+
5+
Windows x64 Release, MSVC v145, Boost 1.92. The benchmark generates deterministic metadata and passes it through the actual SubscriberInstance metadata parser and configuration builder. Its timer excludes XML generation, compression, network transfer, and post-parse validation.
6+
7+
480,000 analog measurements concentrated on eight devices (60,000 each), 185,450,687 XML bytes:
8+
9+
| Stage | Original implementation | Indexed frame construction |
10+
|---|---:|---:|
11+
| Complete metadata processing | 107.081 s | 10.947 s |
12+
| Configuration frames | 92.934 s | 0.348 s |
13+
| XML parsing | 0.504 s | 0.369 s |
14+
15+
A final run after widening metadata indexes completed in **9.783 s**, with all output validation passing. These are individual runs and vary with machine load. Three alternating 50,000-point comparisons gave median totals of 5.479 s original and 1.211 s optimized.
16+
17+
This reproduces a substantial client-side delay locally and identifies configuration construction as the dominant cost for this shape. It does not establish the cause of the client's reported 2.5-minute connection delay: their metadata distribution, server work, transfer, and hardware remain unmeasured.
18+
19+
## Fix
20+
21+
ConstructConfigurationFrames previously scanned a device's measurements for every analog/digital index and scanned its phasors for every phasor index. It now builds lookup vectors once per device. Ordering, missing-index placeholders, and first-match handling of duplicate indexes are preserved.
22+
23+
SignalReference.Index, MeasurementMetadata.PhasorSourceIndex, and PhasorMetadata.SourceIndex now use signed int32_t. Parsing, helper signatures, publisher phasor maps, and metadata filtering preserve these values. Frame loops use int64_t counters so neither 65,535 nor INT32_MAX causes counter wraparound. The aggregate phasor count uses size_t.
24+
25+
Runtime signal-cache and compact-measurement indexes were already int32_t; their wire layout is unchanged. Cache comments have been corrected accordingly.
26+
27+
## Validation
28+
29+
The native benchmark regression mode checks:
30+
31+
- Dense and sparse analog, digital, and phasor metadata, including gzip input.
32+
- Indexes 65,535, 65,536, and 100,000 for each kind; placeholder positions and signal identities.
33+
- Duplicate indexes retain the original first-match behavior.
34+
- A complete 70,000-measurement metadata set.
35+
- A 70,001-entry wire cache and compact-measurement round trips, including INT32_MAX.
36+
- Publisher metadata round trips for indexes 34,464 and 100,000, which collided when narrowed to 16 bits.
37+
- TSSC compressed round trips through indexes 65,535, 65,536, and 100,000.
38+
39+
Build and execution instructions: [MetadataBenchmark](../src/samples/MetadataBenchmark/README.md).
40+
41+
Local evidence is under build/startup-diagnostics (ignored build artifacts): baseline-summary.txt, optimized-summary.txt, massive-baseline.out/.log, massive-optimized.out/.log, massive-int32.out/.log, repeat-baseline-*.out/.log, repeat-optimized-*.out/.log, and int32-regressions.out/.log. MetadataBenchmark-baseline.exe preserves the original frame algorithm for local comparisons.
42+
43+
## Limits and integration
44+
45+
- Configuration frames still allocate placeholders through the maximum index. A sparse enormous index can exhaust memory; signed 32-bit fields do not make billions of frame slots practical. INT32_MAX was tested in parsing, cache, and compact records, not by allocating that many configuration slots or TSSC states.
46+
- Per-phasor angle/magnitude association still scans device measurements and can become costly on a device with many phasors. This optimization addresses configuration-frame construction.
47+
- The separately observed duplicate connection initialization remains unchanged; see StartupDiagnostics.md.
48+
- These changes alter the public native metadata ABI. Rebuild all native consumers. Before using this library with net-cppapi, update its three SWIG field declarations, regenerate bindings, and rebuild both managed and native wrapper components together. Existing .NET binaries have not been replaced.
49+
- Boost compatibility uses explicit version checks; pre-1.66 branches were not compiled in this environment.
50+
51+
## Final native live check
52+
53+
After the performance and int32 changes, the rebuilt StartupTiming.exe connected to openHistorian at 127.0.0.1:7175 with metadata enabled. First measurement arrived 143.550 ms after connection; the five-second run received 19,860 measurements at approximately 4,100/sec and exited successfully. Evidence: build/startup-diagnostics/final-live.out and final-live.log. This checks compatibility with the local publisher, whose small metadata set does not exercise the large-index cases covered by the synthetic regressions.
54+
55+
Production correctness tests are also available in [ConfigurationFramesTest](../src/samples/ConfigurationFramesTest/README.md), independently of this diagnostics branch.
56+

‎docs/StartupDiagnostics.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Diagnosing connection-to-first-data delays
2+
3+
## Branch scope
4+
5+
This diagnostics branch is based on production commit 5a5edac. Performance and int32 fixes, together with samples/ConfigurationFramesTest, are already in main. This branch adds opt-in tracing, StartupTiming, MetadataBenchmark, and investigation notes. Measurements below were collected during the original investigation unless explicitly marked as branch validation.
6+
7+
8+
## Findings from source inspection
9+
10+
The default SubscriberInstance path is sequential:
11+
12+
connection ready -> request metadata -> receive entire response -> callback queue -> decompress -> parse XML -> construct device/measurement/phasor objects -> construct configuration frames -> install maps -> ParsedMetadata callback -> destroy temporary metadata structures -> send Subscribe -> publisher processes subscription -> receive/decode signal index cache -> receive/decode data -> measurement callback.
13+
14+
`SubscriberInstance::HandleMetadata` waits for `ReceivedMetadata` to return before calling `Subscribe`. The subscription filter does not limit the metadata request: that uses the separate `MetadataFilters` setting. A filter excluding STAT therefore still downloads/processes the default metadata set.
15+
16+
### Findings in the original implementation
17+
18+
1. **Configuration-frame construction.** `ConstructConfigurationFrames` calls `TryFindMeasurement` (a linear scan of the device's measurement vector) for every analog/digital index. `GetSignalKindCount` actually returns the maximum signal-reference index, not the number of records. Cost grows roughly with `measurements-per-device * (maximum-analog-index + maximum-digital-index)`. Dense 50,000 analogs on one device can mean roughly 1.25 billion comparisons. Sparse high indexes also generate placeholder records. The phasor configuration loop repeatedly scans the phasor list. This is per-device scaling, not automatically quadratic in the total metadata count across small devices.
19+
2. **Phasor association.** For each phasor, ReceivedMetadata scans the associated device's measurements to locate angle/magnitude. Approximately `sum(phasors-per-device * measurements-per-device)` in the worst case.
20+
3. **Metadata materialization and memory pressure.** Every record creates objects/strings and parses UpdatedOn using string processing and locale-based streams. The metadata maps are copied into member maps; temporary maps, XML storage, and the expanded payload are then destroyed before Subscribe. Raw XML parsing is only one part of this work.
21+
4. **Transfer/decompression and signal cache.** Metadata and cache decompression use CopyStream, which appends bytes one at a time to a vector. Cache Decode performs per-record allocations/hash insertions, with no pre-reservation. These deserve measurement, but do not show the repeated whole-device scans above.
22+
23+
The client Subscribe method builds and sends a small connection string containing the filter; it does not expand the filter to every measurement locally. The publisher performs selection and subscription setup. A long gap after sending Subscribe and before receiving its response/cache points toward publisher work or transfer, rather than this client's filter evaluation. The C++ publisher code does not establish what openHistorian's publisher spends time doing.
24+
25+
**Fixed:** configuration-frame construction now builds per-device index lookups instead of repeatedly scanning each device. Metadata indexes and publisher mappings use int32_t; frame loops use int64_t counters to avoid wraparound. See [MetadataScaling.md](MetadataScaling.md) for the local reproduction, measurements, and boundary tests. Phasor association remains a separate potential hotspot.
26+
27+
## Opt-in instrumentation
28+
29+
Rebuild the native C++ library/client with the modified sources, then launch from PowerShell:
30+
31+
```powershell
32+
$env:STTP_STARTUP_TRACE = '1'
33+
# Run your rebuilt native client with its normal IP/port arguments, redirecting stderr:
34+
# .\YourClient.exe 127.0.0.1 7175 2> startup-trace.log
35+
```
36+
37+
The metadata field changes alter the native ABI. Update the three corresponding SWIG field declarations to int32_t and regenerate the bindings before rebuilding the wrapper. For the .NET wrapper, rebuild/relink `sttp.net.lib.dll` against these instrumented sources and redeploy it before testing ConnectionTest. Merely republishing the .NET executable does not incorporate these C++ changes. The old prebuilt native DLL will not emit these diagnostics.
38+
39+
Unset STTP_STARTUP_TRACE or set it to 0 to disable tracing. The setting is read once per process. Events go directly to stderr and are flushed; they are independent of StatusMessage overrides and the callback queue. No measurement values or metadata content are logged. Before constructing configuration frames, a diagnostic-only scan reports maximum per-device measurement/phasor counts and maximum analog/digital/phasor reference indexes. Its duration is reported separately.
40+
41+
Log format contains monotonic milliseconds since the first trace event, native DataSubscriber pointer (to separate connections), stage duration, and item/byte count where applicable. Scope labels appear on entry and exit; marks report elapsed time since the previous mark. Logs can interleave across threads, so compare timestamps and subscriber pointers. Diagnostic I/O adds some overhead; redirect to a local file.
42+
43+
| Compare stages | What the interval covers |
44+
|---|---|
45+
| metadata request sending -> metadata response received | Request sending, publisher preparation, response transfer |
46+
| metadata response received -> response queued -> metadata callback scope entry | Payload copying and callback scheduling; queue and callback may overlap |
47+
| metadata decompress/copy complete | Gzip expansion or uncompressed payload copy |
48+
| XML parse complete | pugixml load_buffer_inplace |
49+
| device / measurement objects complete | Record conversion, timestamps, strings and map insertion |
50+
| phasor objects and measurement matching complete | Phasor record conversion and association scans |
51+
| configuration frames complete | Indexed lookups and placeholder construction |
52+
| metadata maps installed | Configuration lock, map copying/replacement |
53+
| ParsedMetadata callback complete -> metadata processing scope exit | Temporary metadata/XML/buffer destruction |
54+
| subscribe command built -> subscribe send returned | Local command construction/send; not publisher completion |
55+
| subscribe send -> acknowledged / signal cache scope entry | Publisher processing and transfer; response ordering may vary |
56+
| cache decompress/copy -> cache decode complete | Cache expansion and decoding |
57+
| first nonempty packet received -> first packet decoded | First packet's decode before measurement callback |
58+
59+
The first-packet marker uses the existing per-subscription receive counter. It is a packet diagnostic, not a guarantee of successful measurements if the packet cannot be decoded or its cache is absent.
60+
61+
## Quick comparison test
62+
63+
Use the same endpoint/filter with one counting subscriber, once with default metadata parsing and once with:
64+
65+
```cpp
66+
subscriber.SetAutoParseMetadata(false); // before Connect/ConnectAsync
67+
```
68+
69+
This skips both metadata retrieval and metadata construction and immediately subscribes with the configured filter. Signal-index-cache decoding is still required. It is suitable for the counting test; code that depends on parsed device/configuration metadata will lose that metadata. A large improvement isolates the metadata path, but does not distinguish server metadata preparation/transfer from client parsing; use the stage timings for that distinction.
70+
71+
Then compare the same subscriber with a small filter versus the full filter while metadata is disabled to assess publisher subscription/cache scaling. Keep compression, endpoint and build configuration fixed. Use Release builds for representative performance and record metadata size, per-device point counts and maximum analog/digital/phasor indexes.
72+
73+
## Native build and measured results
74+
75+
The Windows x64 Release diagnostic client is now available at `build/startup-diagnostics/StartupTiming.exe`. Build and run instructions are in `src/samples/StartupTiming/README.md`. Its `--no-metadata` option runs the comparison above without changing the filter. The native library and client were successfully compiled using MSVC v145 and Boost 1.92.
76+
77+
Required compatibility changes: use Boost's built-in UUID std::hash from 1.86 onward, use modern Asio resolver/address APIs with explicit Boost version checks, and explicitly include chrono in ANTLR's profiling source. The configuration-frame optimization is described in MetadataScaling.md. The pre-1.66 Boost branches have not been compiled in this environment.
78+
79+
Initial five-second tests against openHistorian at 127.0.0.1:7175:
80+
81+
| Mode | Connection to first measurement | Received | Exit code |
82+
|---|---:|---:|---:|
83+
| Metadata enabled | 241.080 ms | 19,584 | 0 |
84+
| Metadata disabled | 35.175 ms | 20,402 | 0 |
85+
86+
Both streamed approximately 4,100 measurements/sec. These are individual diagnostic runs, not benchmark averages. Raw traces and console output are in `build/startup-diagnostics/metadata.log`, `metadata-output.log`, `no-metadata.log`, and `no-metadata-output.log`.
87+
88+
For the first metadata response (50,802 compressed bytes, 448,356 expanded bytes, 747 measurement records across 5 devices), the timings were approximately:
89+
90+
- Metadata request to complete response: 116 ms.
91+
- Decompression/copy: 14.2 ms.
92+
- XML parsing: 0.8 ms.
93+
- Measurement-object construction: 15.6 ms.
94+
- Phasor conversion/matching: 3.6 ms.
95+
- Configuration frames: 0.23 ms.
96+
- Signal cache decompression/decoding: approximately 1 ms combined.
97+
98+
This small metadata set does not reproduce the large-device configuration scaling hypothesis. Its maximum per-device measurement count was 169, with maximum analog/digital/phasor indexes of 14/2/59. A trace from the slow client is still needed to locate the 2.5-minute delay.
99+
100+
### Additional observed defect: duplicate startup requests
101+
102+
The trace shows two `connection ready` events, two metadata requests, and two metadata responses/subscriptions for one connection. With metadata disabled it shows two immediate Subscribe commands. Source inspection confirms that `SubscriberInstance::Connect()` calls `ConnectionEstablished()` and `HandleConnect()`, and the registered `HandleConnectionEstablished()` callback also calls them. This duplicates startup work and can trigger a resubscription while the first stream is starting.
103+
104+
The diagnostic patch deliberately retains this behavior so it can be measured. This finding is specific to the current C++ checkout; it does not prove that the older prebuilt SWIG DLL has the same behavior. The diagnostic client retains the earliest connection callback timestamp so duplicate notifications do not reset its timer.
105+
106+
107+
108+
109+
## Diagnostics branch validation
110+
111+
Rebuilt on codex/startup-diagnostics, based on production commit 5a5edac. With tracing enabled, the local openHistorian check at 127.0.0.1:7175 received first data in 171.366 ms and 19,856 measurements over the five-second run (roughly 4,100/sec). Output and trace: build/review-stages/04-live.out and 04-live.log. Timings are individual runs, not benchmark averages.

‎src/lib/transport/DataSubscriber.cpp‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
//******************************************************************************************************
2525

2626
#include "DataSubscriber.h"
27+
#include "StartupTrace.h"
2728
#include "Constants.h"
2829
#include "CompactMeasurement.h"
2930
#include "../Convert.h"
@@ -386,6 +387,8 @@ void DataSubscriber::HandleSucceeded(const uint8_t commandCode, uint8_t* data, c
386387
// Do not break on these messages because there is
387388
// still an associated message to be processed.
388389
m_subscribed = (commandCode == ServerCommand::Subscribe);
390+
if (m_subscribed)
391+
diagnostics::StartupEvent(this, "subscribe acknowledged");
389392
[[fallthrough]];
390393
case ServerCommand::UpdateProcessingInterval:
391394
case ServerCommand::RotateCipherKeys:
@@ -435,7 +438,9 @@ void DataSubscriber::HandleFailed(const uint8_t commandCode, uint8_t* data, cons
435438
// Handles metadata refresh messages from the server.
436439
void DataSubscriber::HandleMetadataRefresh(const uint8_t* data, const uint32_t offset, const uint32_t length)
437440
{
441+
diagnostics::StartupEvent(this, "metadata response received (bytes)", 0.0, length);
438442
Dispatch(&MetadataDispatcher, data, offset, length);
443+
diagnostics::StartupEvent(this, "metadata response queued");
439444
}
440445

441446
// Handles data start time reported by the server at the beginning of a subscription.
@@ -456,6 +461,7 @@ void DataSubscriber::HandleUpdateSignalIndexCache(const uint8_t* data, uint32_t
456461
if (data == nullptr)
457462
return;
458463

464+
diagnostics::StartupPhase startup(this, "signal cache processing scope");
459465
vector<uint8_t> uncompressedBuffer;
460466
int32_t cacheIndex = 0;
461467

@@ -486,8 +492,10 @@ void DataSubscriber::HandleUpdateSignalIndexCache(const uint8_t* data, uint32_t
486492
WriteBytes(uncompressedBuffer, data, offset, length);
487493
}
488494

495+
startup.Mark("signal cache decompress/copy complete (bytes)", uncompressedBuffer.size());
489496
SignalIndexCachePtr signalIndexCache = NewSharedPtr<SignalIndexCache>();
490497
signalIndexCache->Decode(uncompressedBuffer, m_subscriberID);
498+
startup.Mark("signal cache decode complete");
491499

492500
m_signalIndexCacheMutex.lock();
493501
m_signalIndexCache[cacheIndex].swap(signalIndexCache);
@@ -548,6 +556,9 @@ void DataSubscriber::HandleDataPacket(uint8_t* data, uint32_t offset, const uint
548556

549557
// Read measurement count and gather statistics
550558
const uint32_t count = EndianConverter::ToBigEndian<uint32_t>(data, offset);
559+
const bool firstPacket = m_totalMeasurementsReceived == 0 && count > 0;
560+
if (firstPacket)
561+
diagnostics::StartupEvent(this, "first nonempty data packet received", 0.0, count);
551562
m_totalMeasurementsReceived += count;
552563
offset += 4; //-V112
553564

@@ -569,6 +580,8 @@ void DataSubscriber::HandleDataPacket(uint8_t* data, uint32_t offset, const uint
569580
else
570581
ParseCompactMeasurements(signalIndexCache, data, offset, length, includeTime, info.UseMillisecondResolution, frameLevelTimestamp, measurements);
571582

583+
if (firstPacket)
584+
diagnostics::StartupEvent(this, "first packet decoded; invoking measurement callback", 0.0, measurements.size());
572585
newMeasurementsCallback(this, measurements);
573586
}
574587
}
@@ -1414,6 +1427,7 @@ void DataSubscriber::Subscribe(const SubscriptionInfo& info)
14141427
// Subscribe to publisher in order to start receiving data.
14151428
void DataSubscriber::Subscribe()
14161429
{
1430+
diagnostics::StartupPhase startup(this, "subscribe send scope");
14171431
stringstream connectionStream;
14181432
vector<uint8_t> buffer;
14191433
uint32_t bigEndianConnectionStringSize;
@@ -1488,7 +1502,9 @@ void DataSubscriber::Subscribe()
14881502
for (size_t i = 0; i < connectionStringSize; ++i)
14891503
buffer[5 + i] = connectionStringPtr[i];
14901504

1505+
startup.Mark("subscribe command built (bytes)", bufferSize);
14911506
SendServerCommand(ServerCommand::Subscribe, buffer.data(), 0, bufferSize);
1507+
startup.Mark("subscribe send returned");
14921508

14931509
// Reset TSSC decompresser on successful (re)subscription
14941510
m_tsscLastOOSReportMutex.lock();

0 commit comments

Comments
 (0)