|
| 1 | +# Diagnosing connection-to-first-data delays |
| 2 | + |
| 3 | +## Branch scope |
| 4 | + |
| 5 | +This diagnostics branch is based on production commit 5a5edac. Performance and int32 fixes, together with samples/ConfigurationFramesTest, are already in main. This branch adds opt-in tracing, StartupTiming, MetadataBenchmark, and investigation notes. Measurements below were collected during the original investigation unless explicitly marked as branch validation. |
| 6 | + |
| 7 | + |
| 8 | +## Findings from source inspection |
| 9 | + |
| 10 | +The default SubscriberInstance path is sequential: |
| 11 | + |
| 12 | +connection ready -> request metadata -> receive entire response -> callback queue -> decompress -> parse XML -> construct device/measurement/phasor objects -> construct configuration frames -> install maps -> ParsedMetadata callback -> destroy temporary metadata structures -> send Subscribe -> publisher processes subscription -> receive/decode signal index cache -> receive/decode data -> measurement callback. |
| 13 | + |
| 14 | +`SubscriberInstance::HandleMetadata` waits for `ReceivedMetadata` to return before calling `Subscribe`. The subscription filter does not limit the metadata request: that uses the separate `MetadataFilters` setting. A filter excluding STAT therefore still downloads/processes the default metadata set. |
| 15 | + |
| 16 | +### Findings in the original implementation |
| 17 | + |
| 18 | +1. **Configuration-frame construction.** `ConstructConfigurationFrames` calls `TryFindMeasurement` (a linear scan of the device's measurement vector) for every analog/digital index. `GetSignalKindCount` actually returns the maximum signal-reference index, not the number of records. Cost grows roughly with `measurements-per-device * (maximum-analog-index + maximum-digital-index)`. Dense 50,000 analogs on one device can mean roughly 1.25 billion comparisons. Sparse high indexes also generate placeholder records. The phasor configuration loop repeatedly scans the phasor list. This is per-device scaling, not automatically quadratic in the total metadata count across small devices. |
| 19 | +2. **Phasor association.** For each phasor, ReceivedMetadata scans the associated device's measurements to locate angle/magnitude. Approximately `sum(phasors-per-device * measurements-per-device)` in the worst case. |
| 20 | +3. **Metadata materialization and memory pressure.** Every record creates objects/strings and parses UpdatedOn using string processing and locale-based streams. The metadata maps are copied into member maps; temporary maps, XML storage, and the expanded payload are then destroyed before Subscribe. Raw XML parsing is only one part of this work. |
| 21 | +4. **Transfer/decompression and signal cache.** Metadata and cache decompression use CopyStream, which appends bytes one at a time to a vector. Cache Decode performs per-record allocations/hash insertions, with no pre-reservation. These deserve measurement, but do not show the repeated whole-device scans above. |
| 22 | + |
| 23 | +The client Subscribe method builds and sends a small connection string containing the filter; it does not expand the filter to every measurement locally. The publisher performs selection and subscription setup. A long gap after sending Subscribe and before receiving its response/cache points toward publisher work or transfer, rather than this client's filter evaluation. The C++ publisher code does not establish what openHistorian's publisher spends time doing. |
| 24 | + |
| 25 | +**Fixed:** configuration-frame construction now builds per-device index lookups instead of repeatedly scanning each device. Metadata indexes and publisher mappings use int32_t; frame loops use int64_t counters to avoid wraparound. See [MetadataScaling.md](MetadataScaling.md) for the local reproduction, measurements, and boundary tests. Phasor association remains a separate potential hotspot. |
| 26 | + |
| 27 | +## Opt-in instrumentation |
| 28 | + |
| 29 | +Rebuild the native C++ library/client with the modified sources, then launch from PowerShell: |
| 30 | + |
| 31 | +```powershell |
| 32 | +$env:STTP_STARTUP_TRACE = '1' |
| 33 | +# Run your rebuilt native client with its normal IP/port arguments, redirecting stderr: |
| 34 | +# .\YourClient.exe 127.0.0.1 7175 2> startup-trace.log |
| 35 | +``` |
| 36 | + |
| 37 | +The metadata field changes alter the native ABI. Update the three corresponding SWIG field declarations to int32_t and regenerate the bindings before rebuilding the wrapper. For the .NET wrapper, rebuild/relink `sttp.net.lib.dll` against these instrumented sources and redeploy it before testing ConnectionTest. Merely republishing the .NET executable does not incorporate these C++ changes. The old prebuilt native DLL will not emit these diagnostics. |
| 38 | + |
| 39 | +Unset STTP_STARTUP_TRACE or set it to 0 to disable tracing. The setting is read once per process. Events go directly to stderr and are flushed; they are independent of StatusMessage overrides and the callback queue. No measurement values or metadata content are logged. Before constructing configuration frames, a diagnostic-only scan reports maximum per-device measurement/phasor counts and maximum analog/digital/phasor reference indexes. Its duration is reported separately. |
| 40 | + |
| 41 | +Log format contains monotonic milliseconds since the first trace event, native DataSubscriber pointer (to separate connections), stage duration, and item/byte count where applicable. Scope labels appear on entry and exit; marks report elapsed time since the previous mark. Logs can interleave across threads, so compare timestamps and subscriber pointers. Diagnostic I/O adds some overhead; redirect to a local file. |
| 42 | + |
| 43 | +| Compare stages | What the interval covers | |
| 44 | +|---|---| |
| 45 | +| metadata request sending -> metadata response received | Request sending, publisher preparation, response transfer | |
| 46 | +| metadata response received -> response queued -> metadata callback scope entry | Payload copying and callback scheduling; queue and callback may overlap | |
| 47 | +| metadata decompress/copy complete | Gzip expansion or uncompressed payload copy | |
| 48 | +| XML parse complete | pugixml load_buffer_inplace | |
| 49 | +| device / measurement objects complete | Record conversion, timestamps, strings and map insertion | |
| 50 | +| phasor objects and measurement matching complete | Phasor record conversion and association scans | |
| 51 | +| configuration frames complete | Indexed lookups and placeholder construction | |
| 52 | +| metadata maps installed | Configuration lock, map copying/replacement | |
| 53 | +| ParsedMetadata callback complete -> metadata processing scope exit | Temporary metadata/XML/buffer destruction | |
| 54 | +| subscribe command built -> subscribe send returned | Local command construction/send; not publisher completion | |
| 55 | +| subscribe send -> acknowledged / signal cache scope entry | Publisher processing and transfer; response ordering may vary | |
| 56 | +| cache decompress/copy -> cache decode complete | Cache expansion and decoding | |
| 57 | +| first nonempty packet received -> first packet decoded | First packet's decode before measurement callback | |
| 58 | + |
| 59 | +The first-packet marker uses the existing per-subscription receive counter. It is a packet diagnostic, not a guarantee of successful measurements if the packet cannot be decoded or its cache is absent. |
| 60 | + |
| 61 | +## Quick comparison test |
| 62 | + |
| 63 | +Use the same endpoint/filter with one counting subscriber, once with default metadata parsing and once with: |
| 64 | + |
| 65 | +```cpp |
| 66 | +subscriber.SetAutoParseMetadata(false); // before Connect/ConnectAsync |
| 67 | +``` |
| 68 | + |
| 69 | +This skips both metadata retrieval and metadata construction and immediately subscribes with the configured filter. Signal-index-cache decoding is still required. It is suitable for the counting test; code that depends on parsed device/configuration metadata will lose that metadata. A large improvement isolates the metadata path, but does not distinguish server metadata preparation/transfer from client parsing; use the stage timings for that distinction. |
| 70 | + |
| 71 | +Then compare the same subscriber with a small filter versus the full filter while metadata is disabled to assess publisher subscription/cache scaling. Keep compression, endpoint and build configuration fixed. Use Release builds for representative performance and record metadata size, per-device point counts and maximum analog/digital/phasor indexes. |
| 72 | + |
| 73 | +## Native build and measured results |
| 74 | + |
| 75 | +The Windows x64 Release diagnostic client is now available at `build/startup-diagnostics/StartupTiming.exe`. Build and run instructions are in `src/samples/StartupTiming/README.md`. Its `--no-metadata` option runs the comparison above without changing the filter. The native library and client were successfully compiled using MSVC v145 and Boost 1.92. |
| 76 | + |
| 77 | +Required compatibility changes: use Boost's built-in UUID std::hash from 1.86 onward, use modern Asio resolver/address APIs with explicit Boost version checks, and explicitly include chrono in ANTLR's profiling source. The configuration-frame optimization is described in MetadataScaling.md. The pre-1.66 Boost branches have not been compiled in this environment. |
| 78 | + |
| 79 | +Initial five-second tests against openHistorian at 127.0.0.1:7175: |
| 80 | + |
| 81 | +| Mode | Connection to first measurement | Received | Exit code | |
| 82 | +|---|---:|---:|---:| |
| 83 | +| Metadata enabled | 241.080 ms | 19,584 | 0 | |
| 84 | +| Metadata disabled | 35.175 ms | 20,402 | 0 | |
| 85 | + |
| 86 | +Both streamed approximately 4,100 measurements/sec. These are individual diagnostic runs, not benchmark averages. Raw traces and console output are in `build/startup-diagnostics/metadata.log`, `metadata-output.log`, `no-metadata.log`, and `no-metadata-output.log`. |
| 87 | + |
| 88 | +For the first metadata response (50,802 compressed bytes, 448,356 expanded bytes, 747 measurement records across 5 devices), the timings were approximately: |
| 89 | + |
| 90 | +- Metadata request to complete response: 116 ms. |
| 91 | +- Decompression/copy: 14.2 ms. |
| 92 | +- XML parsing: 0.8 ms. |
| 93 | +- Measurement-object construction: 15.6 ms. |
| 94 | +- Phasor conversion/matching: 3.6 ms. |
| 95 | +- Configuration frames: 0.23 ms. |
| 96 | +- Signal cache decompression/decoding: approximately 1 ms combined. |
| 97 | + |
| 98 | +This small metadata set does not reproduce the large-device configuration scaling hypothesis. Its maximum per-device measurement count was 169, with maximum analog/digital/phasor indexes of 14/2/59. A trace from the slow client is still needed to locate the 2.5-minute delay. |
| 99 | + |
| 100 | +### Additional observed defect: duplicate startup requests |
| 101 | + |
| 102 | +The trace shows two `connection ready` events, two metadata requests, and two metadata responses/subscriptions for one connection. With metadata disabled it shows two immediate Subscribe commands. Source inspection confirms that `SubscriberInstance::Connect()` calls `ConnectionEstablished()` and `HandleConnect()`, and the registered `HandleConnectionEstablished()` callback also calls them. This duplicates startup work and can trigger a resubscription while the first stream is starting. |
| 103 | + |
| 104 | +The diagnostic patch deliberately retains this behavior so it can be measured. This finding is specific to the current C++ checkout; it does not prove that the older prebuilt SWIG DLL has the same behavior. The diagnostic client retains the earliest connection callback timestamp so duplicate notifications do not reset its timer. |
| 105 | + |
| 106 | + |
| 107 | + |
| 108 | + |
| 109 | +## Diagnostics branch validation |
| 110 | + |
| 111 | +Rebuilt on codex/startup-diagnostics, based on production commit 5a5edac. With tracing enabled, the local openHistorian check at 127.0.0.1:7175 received first data in 171.366 ms and 19,856 measurements over the five-second run (roughly 4,100/sec). Output and trace: build/review-stages/04-live.out and 04-live.log. Timings are individual runs, not benchmark averages. |
0 commit comments