Refactor TSDataType switches to use TypeService - #18606
Open
jt2594838 wants to merge 112 commits into
Open
Conversation
Caideyipi
requested changes
Sep 9, 2026
Caideyipi
left a comment
Caideyipi
left a comment
Collaborator
There was a problem hiding this comment.
发现 4 个会导致功能错误或序列化不兼容的问题，建议修复后再合并。下面的行级评论分别说明了复现测试和修复方向。
|
|
||
| @Override | ||
| public void initializeFloatValues() { | ||
| floatValues = new FloatBigArray(Float.MIN_VALUE); |
Collaborator
There was a problem hiding this comment.
Float.MIN_VALUE and Double.MIN_VALUE are the smallest positive values, not lower bounds. With this initialization, MAX ignores zero and all negative FLOAT/DOUBLE inputs, leaving inits false and returning NULL. The existing CI tests testFloatMaxWithNonPositiveInput, testDoubleMaxWithNonPositiveInput, and testDoubleMaxWithNonPositiveIntermediateInput reproduce this regression. Please keep the previous NEGATIVE_INFINITY initialization or use another true lower bound.
| } | ||
|
|
||
| public void setXResult(final TsPrimitiveType result, final Column column, final int index) { | ||
| type.setTo(result, column, index); |
Collaborator
There was a problem hiding this comment.
Type.setTo(TsPrimitiveType, Column, int) copies the primitive into the column; it does not copy the column value into the primitive. This call therefore leaves xResult at its default value and can overwrite the input column. CI reproduces this in AccumulatorTest.maxByAccumulatorTest (-99 expected, 0 returned). Please use the correct column-to-primitive copy operation.
| default: | ||
| throw new UnSupportedDataTypeException(String.format(DATATYPE_UNSUPPORTED, dataType)); | ||
| } | ||
| Type.fromTsDataType(dataType).serializeArray(column, rowCount, buffer); |
Collaborator
There was a problem hiding this comment.
Type.serializeArray uses TsFile's binary-array format, which writes a presence byte before each binary value. The Tablet reader in RAW_ARRAY_BYTE_BUFFER_DESERIALIZER_SERVICE still reads the legacy Tablet format (length + payload only, with nulls represented by the separate bitmap), so the first presence byte is interpreted as part of the length and the stream becomes misaligned. CI fails with BufferUnderflowException in InsertMultiTabletsNodeSerdeTest. Please preserve the existing Tablet wire format or update both serialization, size calculation, and all readers together.
| break; | ||
| } | ||
| return size; | ||
| return Type.fromTsDataType(dataType).serializedSize(column, start, end); |
Collaborator
There was a problem hiding this comment.
The WAL size calculation now uses Type.serializedSize, whose binary-array size includes a presence byte, but WAL_ARRAY_WRITER_SERVICE still writes only the length and payload. The calculated WAL entry size therefore disagrees with the bytes actually written, which breaks WAL offsets/search indexes. The three WALFileTest failures in CI reproduce this. Please use a size calculation matching the WAL writer's wire format.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The architecture check is not repository-wide, and several new localized-message definitions violate established naming and localization requirements.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Refactors type-specific logic into TsFile Type/TypeService abstractions across UDFs, query execution, storage, clients, and examples.
Changes:
- Replaces widespread
TSDataTypeswitches with centralized type services. - Updates TsFile APIs/version and tablet/WAL-related handling.
- Adds architecture and regression tests plus localized messages.
File summaries
| File | Description |
|---|---|
pom.xml |
Updates TsFile snapshot. |
library-udf/pom.xml |
Adds TsFile dependency. |
library-udf/.../util/Util.java |
Delegates row value operations. |
library-udf/.../match/UDAFPatternMatch.java |
Uses numeric column reader. |
library-udf/.../match/UDAFDTWMatch.java |
Uses numeric column reader. |
library-udf/.../drepair/UDTFValueRepair.java |
Delegates repaired-value output. |
library-udf/.../drepair/UDTFValueFill.java |
Delegates filled-value output. |
library-udf/.../drepair/UDTFTimestampRepair.java |
Delegates cast output. |
library-udf/.../anomaly/UDTFTwoSidedFilter.java |
Delegates filtered output. |
library-udf/.../i18n/en/LibraryUdfMessages.java |
Adds English messages. |
library-udf/.../i18n/zh/LibraryUdfMessages.java |
Adds Chinese messages. |
node-commons/.../InternalTypeManagerTest.java |
Tests type conversions. |
node-commons/.../MasterRepairUtil.java |
Uses generic numeric row access. |
node-commons/.../UDTFValueTrend.java |
Delegates previous-value reads. |
node-commons/.../UDTFValueDifference.java |
Selects difference operators. |
node-commons/.../UDTFTopK.java |
Delegates queue construction. |
node-commons/.../UDTFNonNegativeValueDifference.java |
Delegates difference calculation. |
node-commons/.../UDTFNonNegativeDerivative.java |
Delegates derivative calculation. |
node-commons/.../UDTFM4.java |
Delegates window transformation. |
node-commons/.../UDTFEqualSizeBucketRandomSample.java |
Delegates row collection. |
node-commons/.../UDTFEqualSizeBucketM4Sample.java |
Delegates M4 sampling. |
node-commons/.../UDTFEqualSizeBucketAggSample.java |
Delegates bucket aggregation. |
node-commons/.../UDTFDerivative.java |
Initializes derivative services. |
node-commons/.../UDTFCommonValueDifference.java |
Uses shared difference operator. |
node-commons/.../UDTFCommonDerivative.java |
Uses shared derivative operator. |
node-commons/.../InternalTypeManager.java |
Simplifies type mapping. |
datanode/.../TSDataTypeSwitchArchitectureTest.java |
Adds switch architecture check. |
datanode/.../DescFakedSeriesReader.java |
Uses TsFile type factory. |
datanode/.../AscFakedSeriesReader.java |
Uses TsFile type factory. |
datanode/.../PrimitiveMemTableTest.java |
Updates primitive test values. |
datanode/.../CompactionCheckerUtils.java |
Updates compaction test values. |
datanode/.../OpcUaNameSpaceMetadataTest.java |
Updates OPC UA test values. |
datanode/.../IoTDBOpcUaClientTest.java |
Updates OPC UA client values. |
datanode/.../TypeInferenceUtils.java |
Delegates auto-cast checks. |
datanode/.../TimeValuePairUtils.java |
Delegates value copying/factories. |
datanode/.../EncodingInferenceUtils.java |
Delegates encoding selection. |
datanode/.../LongTVList.java |
Uses typed primitive creation. |
datanode/.../IntTVList.java |
Uses typed primitive creation. |
datanode/.../FloatTVList.java |
Uses typed primitive creation. |
datanode/.../DoubleTVList.java |
Uses typed primitive creation. |
datanode/.../BooleanTVList.java |
Uses typed primitive creation. |
datanode/.../BinaryTVList.java |
Uses typed primitive creation. |
datanode/.../TsFileSplitTool.java |
Delegates chunk writes. |
datanode/.../TsFileSplitByPartitionTool.java |
Delegates partitioned writes. |
datanode/.../LoadTsFileManager.java |
Updates last-value conversion. |
datanode/.../TsFileResourceUtils.java |
Updates resource last values. |
datanode/.../MemAlignedPageReader.java |
Delegates statistics updates. |
datanode/.../SingleSeriesCompactionExecutor.java |
Delegates compaction writes. |
datanode/.../ReadChunkAlignedSeriesCompactionExecutor.java |
Delegates size estimation. |
datanode/.../transformation/dag/util/TypeUtils.java |
Centralizes column operations. |
datanode/.../ElasticSerializableRowRecordListBackedMultiColumnRow.java |
Widens numeric reads. |
datanode/.../InsertRowStatement.java |
Delegates value deserialization. |
datanode/.../OperatorTreeGenerator.java |
Delegates constant fills. |
datanode/.../OperatorGeneratorUtil.java |
Delegates value-size estimates. |
datanode/.../WindowManagerFactory.java |
Delegates window creation. |
datanode/.../LastQueryAggTableScanOperator.java |
Delegates primitive cloning. |
datanode/.../TreeInsertTabletStatementGenerator.java |
Uses type converter directly. |
datanode/.../TransformOperator.java |
Delegates column writes. |
datanode/.../TableInsertTabletStatementGenerator.java |
Uses type converter directly. |
datanode/.../AggregationUtil.java |
Delegates output-size estimates. |
datanode/.../AccumulatorFactory.java |
Delegates mode accumulator creation. |
datanode/.../PipeMemoryWeightUtil.java |
Uses typed size estimates. |
datanode/.../TimeSeriesRuntimeState.java |
Adds typed row dispatch. |
datanode/.../SinglePageWholeChunkReader.java |
Delegates memory estimates. |
datanode/.../PipeTabletUtils.java |
Delegates tablet value insertion. |
datanode/.../PipeRow.java |
Delegates object reads. |
datanode/.../PipeDataTypeTransformer.java |
Delegates pipe type conversion. |
datanode/.../IoTDBDescriptor.java |
Delegates default encoding. |
datanode/.../IoTDBConfig.java |
Implements encoding provider. |
datanode/.../i18n/zh/StorageEngineMessages.java |
Normalizes whitespace. |
datanode/.../i18n/zh/DataNodeQueryMessages.java |
Adds Chinese query messages. |
datanode/.../i18n/zh/DataNodePipeMessages.java |
Adds Chinese pipe message. |
datanode/.../i18n/zh/DataNodeMiscMessages.java |
Adds Chinese type messages. |
datanode/.../i18n/en/DataNodeQueryMessages.java |
Adds English query messages. |
datanode/.../i18n/en/DataNodePipeMessages.java |
Adds English pipe message. |
datanode/.../i18n/en/DataNodeMiscMessages.java |
Adds English type messages. |
iotdb-core/datanode/pom.xml |
Adds ASM test dependency. |
calc-commons/.../SerializableTVList.java |
Delegates row memory sizing. |
calc-commons/.../TryCastFunctionColumnTransformer.java |
Reuses cast dispatch. |
calc-commons/.../RoundColumnTransformer.java |
Uses generic numeric access. |
calc-commons/.../CastFunctionColumnTransformer.java |
Reuses cast dispatch. |
calc-commons/.../*GreatestColumnTransformer.java |
Exposes constructors to services. |
calc-commons/.../*LeastColumnTransformer.java |
Exposes constructors to services. |
calc-commons/.../AbstractGreatestLeastColumnTransformer.java |
Delegates transformer selection. |
calc-commons/.../MergeSortFullOuterJoinOperator.java |
Uses typed row functions. |
calc-commons/.../TableRegressionAccumulator.java |
Simplifies numeric reads. |
calc-commons/.../TableCovarianceAccumulator.java |
Simplifies numeric reads. |
calc-commons/.../TableCorrelationAccumulator.java |
Simplifies numeric reads. |
calc-commons/.../RateFunctionValidation.java |
Delegates numeric conversion. |
calc-commons/.../GroupedRegressionAccumulator.java |
Simplifies grouped reads. |
calc-commons/.../GroupedCovarianceAccumulator.java |
Simplifies grouped reads. |
calc-commons/.../GroupedCorrelationAccumulator.java |
Simplifies grouped reads. |
calc-commons/.../LongBigArray.java |
Adds byte serialization. |
calc-commons/.../IntBigArray.java |
Adds byte serialization. |
calc-commons/.../FloatBigArray.java |
Adds byte serialization. |
calc-commons/.../DoubleBigArray.java |
Adds byte serialization. |
calc-commons/.../BooleanBigArray.java |
Adds byte serialization. |
calc-commons/.../BinaryBigArray.java |
Adds byte serialization. |
calc-commons/.../AbstractApproxMostFrequentAccumulator.java |
Updates localized message key. |
calc-commons/.../ColumnList.java |
Adds cross-column equality. |
calc-commons/.../ValueWindowFunction.java |
Centralizes default-value writes. |
calc-commons/.../LeadFunction.java |
Removes local type dispatch. |
calc-commons/.../LagFunction.java |
Removes local type dispatch. |
calc-commons/.../MergeSortComparator.java |
Delegates comparator creation. |
calc-commons/.../JoinKeyComparatorFactory.java |
Delegates join comparison. |
calc-commons/.../RegressionAccumulator.java |
Simplifies numeric reads. |
calc-commons/.../CovarianceAccumulator.java |
Simplifies numeric reads. |
calc-commons/.../CorrelationAccumulator.java |
Simplifies numeric reads. |
calc-commons/.../CentralMomentAccumulator.java |
Uses numeric converter service. |
session/.../SessionUtilsTest.java |
Tests mixed value encoding. |
service-rpc/.../PreparedParameterSerdeTest.java |
Tests unsupported types. |
service-rpc/.../PreparedParameterSerde.java |
Delegates parameter decoding. |
jdbc/.../GroupedLSBWatermarkEncoder.java |
Adds typed watermark encoding. |
isession/.../TypeServices.java |
Adds RPC field readers. |
isession/.../SessionDataSet.java |
Uses RPC field readers. |
udf-api/.../RowImpl.java |
Widens numeric reads and mapping. |
udf-api/.../access/Row.java |
Documents numeric conversion. |
integration-test/.../TwoSum.java |
Delegates example arithmetic. |
rest/.../FastLastHandlerTest.java |
Updates primitive test values. |
example/udf/pom.xml |
Adds provided TsFile dependency. |
example/session/.../TabletExample.java |
Adds typed CSV parsers. |
Review details
- Files reviewed: 120/299 changed files
- Comments generated: 11
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+48
to
+49
| throw new UnSupportedDataTypeException( | ||
| String.format("Data type %s is not supported.", type.getTypeEnum())); |
Comment on lines
+606
to
+609
| public static final String DATA_TYPE_NOT_CONSISTENT_FMT = | ||
| "data type is not consistent, input %s, registered %s"; | ||
| public static final String DATA_TYPE_NOT_CONSISTENT_WITH_CAUSE_FMT = | ||
| "data type is not consistent, input %s, registered %s because %s"; |
Comment on lines
+1547
to
+1548
| public static final String UNSUPPORTED_DATA_TYPE_FOR_COLUMN_FMT = | ||
| "unsupported data type %s for column %s"; |
Comment on lines
+531
to
+534
| public static final String VALUE_CANNOT_BE_CAST_TO_DATA_TYPE_FMT = | ||
| "\"%s\" cannot be cast to [%s]"; | ||
| public static final String UNSUPPORTED_DATA_TYPE_FMT = | ||
| "Unsupported data type %s"; |
Comment on lines
+1035
to
+1036
| public static final String UNSUPPORTED_SCALAR_SUBQUERY_RESULT_DATA_TYPE_FMT = | ||
| "Unsupported data type for scalar subquery result: %s"; |
Comment on lines
+35
to
+37
| // Avoid building the entire server dependency graph: only switch owners need checking. | ||
| .withImportOption(location -> TSDataTypeSwitchRule.hasSwitch(location.asURI())) | ||
| .importPackages("org.apache.iotdb")); |
Comment on lines
+63
to
+65
| public static final String FREQUENCY_MUST_BE_POSITIVE = "The param 'frequency' must > 0."; | ||
| public static final String AMPLIFICATION_MUST_BE_AT_LEAST_1 = | ||
| "The param 'amplification' must >= 1."; |
Comment on lines
+63
to
+65
| public static final String FREQUENCY_MUST_BE_POSITIVE = "The param 'frequency' must > 0."; | ||
| public static final String AMPLIFICATION_MUST_BE_AT_LEAST_1 = | ||
| "The param 'amplification' must >= 1."; |
Comment on lines
+106
to
+110
| // ExactOrderStatistics | ||
| public static final String UNSUPPORTED_DATA_TYPE = "Unsupported data type: %s"; | ||
|
|
||
| // UDAFQuantile | ||
| public static final String UNSUPPORTED_DATA_TYPE_IN_QUANTILE = "Unsupported data type"; |
Comment on lines
+111
to
+114
| public static final String QUANTILE_K_MUST_BE_AT_LEAST_100 = | ||
| "Size K has to be greater than or equal to 100."; | ||
| public static final String QUANTILE_RANK_MUST_BE_IN_RANGE = | ||
| "rank has to be greater than 0 and less than or equal to 1."; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is changed
Validation