TSDB decoding consistency#2757
Conversation
…DB storage - Add TIMESTAMP and RX_TIMESTAMP to Packet.RESERVED_ITEM_NAMES to prevent collisions with QuestDB table columns - Add validation in PacketItemParser to reject reserved item names - Store received_time in telemetry_decom_topic message for direct access - Optimize TSDB storage by skipping redundant time columns: - PACKET_TIMESECONDS, PACKET_TIMEFORMATTED (derived from timestamp) - RECEIVED_TIMESECONDS, RECEIVED_TIMEFORMATTED (derived from rx_timestamp) - Read received_time directly from topic message instead of json_data - Update tests to reflect new behavior Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Creates common QuestDBClient decode utilities in both Ruby and Python to properly decode JSON-encoded arrays/objects and base64-encoded binary data when reading from QuestDB. Updates: - New openc3/lib/openc3/utilities/questdb_client.rb with decode_value and decode_hash methods - Added decode_value and decode_dict static methods to questdb_client.py - Updated cvt_model.rb to decode values in tsdb_lookup - Updated cvt_model.py to decode values in tsdb_lookup - Updated logged_streaming_thread.rb to decode values in stream_items and build_packet_entry methods Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Store 64-bit INT/UINT columns as DECIMAL(20,0) instead of VARCHAR for ~20% storage savings (16 bytes vs ~20+ bytes per value) - Send integers as strings via ILP (QuestDB casts to DECIMAL) - Decode DECIMAL values back to integers using data_type parameter - Add type-aware decode_value with explicit data_type and array_size - Sanitize additional ILP protocol special characters (=!@#$^&) - Fix rx_timestamp to use datetime (timestamp) instead of TimestampNanos - Handle BigDecimal in Ruby and Decimal in Python for round-tripping Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds comprehensive integration tests that verify data can be written to and read from QuestDB correctly across both Python and Ruby implementations. Includes GitHub Actions workflow to run tests automatically with a QuestDB service container on changes to TSDB-related files. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2757 +/- ##
==========================================
- Coverage 79.16% 78.68% -0.48%
==========================================
Files 670 671 +1
Lines 54259 54738 +479
Branches 731 731
==========================================
+ Hits 42954 43073 +119
- Misses 11225 11585 +360
Partials 80 80
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Add run_roundtrip_test helper to Python tests (942 → 398 lines) - Add run_roundtrip_test helper to Ruby CvtModel tests (736 → 326 lines) - Add run_streaming_test helper to Ruby streaming tests - Expand streaming tests to cover all data types (INT 8/16/32/64-bit, UINT 8/16/32/64-bit, FLOAT 32/64-bit, STRING, BLOCK, DERIVED, arrays, objects) matching the CvtModel test coverage Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
QuestDB stores float special values (inf, -inf, nan) as NULL, which loses the original value. This adds sentinel values near float max/min to preserve these values through storage and retrieval. Implements: - 64-bit and 32-bit sentinel constants for both Python and Ruby - Python encode_float_special_values() for writing with proper bit size - Python/Ruby decode_float_special_values() for reading back - Tracks float column bit sizes in Python QuestDBClient for encoding Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
![aikido-pr-checks[bot]](https://avatars.githubusercontent.com/in/898896?s=60&v=4){kind=small}
| python-version: "3.12" | ||
|
|
||
| - name: Set up Ruby | ||
| uses: ruby/setup-ruby@v1 |
There was a problem hiding this comment.
3rd party Github Actions should be pinned - medium severity
A third-party GitHub Action was imported, and is not pinned via a hash. This leaves your CI/CD at risk for potential supply chain attacks, if the affected GitHub Action is compromised.
Show Remediation
Remediation - high confidence
This patch mitigates a potential supply chain attack by pinning the version of third-party Github Actions to their commit SHA.
| uses: ruby/setup-ruby@v1 | |
| uses: ruby/setup-ruby@90be1154f987f4dc0fe0dd0feedac9e473aa4ba8 # v1.286.0 |
| OPENC3_DEVEL: ${GITHUB_WORKSPACE}/openc3 | ||
|
|
||
| - name: Install Poetry | ||
| uses: snok/install-poetry@v1 |
There was a problem hiding this comment.
3rd party Github Actions should be pinned - medium severity
A third-party GitHub Action was imported, and is not pinned via a hash. This leaves your CI/CD at risk for potential supply chain attacks, if the affected GitHub Action is compromised.
Show Remediation
Remediation - high confidence
This patch mitigates a potential supply chain attack by pinning the version of third-party Github Actions to their commit SHA.
| uses: snok/install-poetry@v1 | |
| uses: snok/install-poetry@76e04a911780d5b312d89783f7b1cd627778900a # v1.4.1 |
- Add support for calculated timestamp items (PACKET_TIMESECONDS, PACKET_TIMEFORMATTED, RECEIVED_TIMESECONDS, RECEIVED_TIMEFORMATTED) in tsdb_lookup for both Ruby and Python cvt_model and logged_streaming_thread - Move shared TSDB code to questdb_client: - TIMESTAMP_ITEMS constant - column_suffix_for_value_type() method - pg_timestamp_to_utc() method for timezone conversion - format_timestamp() method for seconds/ISO 8601 formatting - Fix Ruby 3.0+ keyword argument handling in test files - Add comprehensive timestamp tests to all three test suites Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
|
||
| # Look up item type info from packet definition | ||
| cache_key = [tgt, pkt] | ||
| unless packet_cache.key?(cache_key) |
There was a problem hiding this comment.
Does this cache ever clear? Needs to invalidate if the target is updated.
There was a problem hiding this comment.
This is a local cache in the stream_items method. Each call requests a new TargetModel.packet but only once so we're not constantly going to redis.
| safe_item_name = QuestDBClient.sanitize_column_name(orig_item_name) | ||
|
|
||
| # Look up item type info from packet definition | ||
| cache_key = [target_name, packet_name] |
There was a problem hiding this comment.
Same cache question. When does it invalidate?
There was a problem hiding this comment.
This is a local cache for the tsdb_lookup() call. It is only cached in each call. But this can still save a bunch of lookup time since we only have to grab the packet definition once.
|
|
||
| # Look up item type info from packet definition | ||
| cache_key = (target_name, packet_name) | ||
| if cache_key not in packet_cache: |
There was a problem hiding this comment.
This is a local cache for the tsdb_lookup() call. It is only cached in each call. But this can still save a bunch of lookup time since we only have to grab the packet definition once.
…SECONDS Rename the designated timestamp columns in QuestDB tables from 'timestamp' and 'rx_timestamp' to 'PACKET_TIMESECONDS' and 'RECEIVED_TIMESECONDS'. This reduces reserved column names by reusing existing reserved item names. - Update table creation to use new column names - Update TIMESTAMP_ITEMS to only contain TIMEFORMATTED items (calculated) - Add handling for stored timestamp items with ns-to-seconds conversion - Update WHERE clauses and query building for new column names - Update tests and mock data for new column names Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
3 participants
No description provided.