Release 2.0.2

Thanks to our community users and developers, 489 improvements and bug fixes have been made in Doris 2.0.2.

Behavior Changes

• Remove json -> operator convert to json_extract #24679

  Remove json '->' operator since it is conflicted with lambda function syntax. It's a syntax sugar for function json_extract and can be replaced with the former.

• Start the script to set metadata_failure_recovery #24308

  Move metadata_failure_recovery from fe.conf to start_fe.sh argument to prevent being used unexpectedly.

• Change ordinary type null value is \N,complex type null value is null #24207

• Optimize priority_ network matching logic for be #23795

• Fix cancel load failed because Job could not be cancelled… #17730

  Allow cancel a retrying load job.

Improvements

Easier to use

• Support custom lib dir to save custom libs #23887

  Add a custom_lib dir to allow users place custom lib files and custom_lib will not be replaced.

• Optimize priority_ network matching logic #23784

  Optimize priority_network logic to avoid error when this config is wrong or not configured.

• Row policy support role #23022

  Support role based auth for row policy.

New optimizer Nereids statistics collection improvement

• Disable file cache while running analysis tasks. #23663
• Show column stats even when error occurred. #23703
• Support basic jdbc external table stats collection. #23965
• Skip unknown col stats check on __internal_scheam and information_schema #24625

Better support for JDBC, HDFS, Hive, MySQL, Max Compute, Multi-Catalog

• Support hadoop viewfs. #24168

• Avoid calling checksum when replaying creating jdbc catalog and fix ranger issue #22369

• Optimize the JDBC Catalog connection error message #23868

  Improve property check and error message for JDBC catalog

• Fix mc decimal type parse, fix wrong obj location #24242

  Fix some issues for Max Compute catalog

• Support sql cache for hms catalog #23391

  SQL cache for Hive catalog

• Merge hms partition events. #22869

  Improve performance for Hive metadata sync

• Add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql #22702

Performance for inverted index query

• Add bkd index query cache to improve perf #23952
• Improve performance for count on index other than match #24678
• Improve match performance without index #24751
• Optimize multiple terms conjunction query #23871 Improve performance of MATCH_ALL
• Optimize unnecessary conversions #24389 Improve performance of MATCH

Improve Array functions

• [Fix old optimizer with some array literal functions #23630
• Improve array union support multi params #24327
• Improve explode func with array nested complex type #24455

Important Bug fixes

• The parameter positions of timestamp diff function to sql are reversed #23601
• Fix old optimizer with some array literal functions #23630
• Fix query cache returns wrong result after deleting partitions. #23555
• Fix potential data loss when clone task's dst tablet is cooldown replica #17644
• Fix array map batch append data with right next_array_item_rowid #23779
• Fix or to in rule #23940
• Fix 'char' function's toSql implementation is wrong #23860
• Record wrong best plan properties #23973
• Make TVF's distribution spec always be RANDOM #24020
• External scan use STORAGE_ANY instead of ANY as distibution #24039
• Runtimefilter target is not SlotReference #23958
• mv in select materialized_view should disable show table #24104
• Fail over to remote file reader if local cache failed #24097
• Fix revoke role operation cause fe down #23852
• Handle status code correctly and add a new error code ENTRY_NOT_FOUND #24139
• Fix leaky abstraction and shield the status code END_OF_FILE from upper layers #24165
• Fix bug that Read garbled files caused be crash. #24164
• Fix be core when user sepcified empty column_separator using hdfs tvf #24369
• Fix need to restart BE after replacing the jar package in java-udf #24372
• Need to call 'set_version' in nested functions #24381
• windown_funnel compatibility issue with multi backends #24385
• correlated anti join shouldn't be translated to null aware anti join #24290
• Change ordinary type null value is \N,complex type null value is null #24207
• Fix analyze failed when there are thousands of partitions. #24521
• Do not use enum as the data type for JavaUdfDataType. #24460
• Fix multi window projection issue temporarily #24568
• Make metadata compatible with 2.0.3 #24610
• Select outfile column order is wrong #24595
• Incorrect result of semi/anti mark join #24616
• Fix broker read issue #24635
• Skip unknown col stats check on __internal_scheam and information_schema #24625
• Fixed bug when parsing multi-character delimiters. #24572
• Fix timezone parse when there is no tzfile #24578
• We need to issue an error when starting FE without setting the Java home environment #23943
• Enable_unique_key_partial_update should be forwarded to master #24697
• Fix paimon file catalog meta issue and replication num analysis issue #24681
• Add more log for ingest_binlog && Fix ingest_binlog not rewrite rowset_meta tablet_uid #24617
• Do not abort when a disk is broken #24692
• colocate join could not work well on full outer join #24700
• Optimize unnecessary conversions #24389
• Optimize the reading efficiency of nullable (string) columns. #24698
• Fix segment cache core when output rowset is nullptr #24778
• Fix duplicate key in schema change #24782
• Make metadata compatible for future version after 2.0.2 #24800
• Fix map/array deserialize string with quote pair #24808
• Failed on arm platform, with clang compiler and pch on, close #24633 #24636
• Table column order is changed if add a column and do truncate #24981
• Make parser mode coarse grained by default #24949

See the complete list of improvements and bug fixes on github .

Big Thanks

Thanks all who contribute to this release:

@adonis0147 @airborne12 @amorynan @AshinGau @BePPPower @BiteTheDDDDt @bobhan1 @ByteYue @caiconghui @CalvinKirs @cambyzju @ChengDaqi2023 @ChinaYiGuan @CodeCooker17 @csun5285 @dataroaring @deadlinefen @DongLiang-0 @Doris-Extras @dutyu @eldenmoon @englefly @freemandealer @Gabriel39 @gnehil @GoGoWen @gohalo @HappenLee @hello-stephen @HHoflittlefish777 @hubgeter @hust-hhb @ixzc @JackDrogon @jacktengg @jackwener @Jibing-Li @JNSimba @kaijchen @kaka11chen @Kikyou1997 @Lchangliang @LemonLiTree @liaoxin01 @LiBinfeng-01 @liugddx @luwei16 @mongo360 @morningman @morrySnow @mrhhsg @Mryange @mymeiyi @neuyilan @pingchunzhang @platoneko @qidaye @realize096 @RYH61 @shuke987 @sohardforaname @starocean999 @SWJTU-ZhangLei @TangSiyang2001 @Tech-Circle-48 @w41ter @wangbo @wsjz @wuwenchi @wyx123654 @xiaokang @XieJiann @xinyiZzz @XuJianxu @xutaoustc @xy720 @xyfsjq @xzj7019 @yiguolei @yujun777 @Yukang-Lian @Yulei-Yang @zclllyybb @zddr @zhangguoqiang666 @zhangstar333 @ZhangYu0123 @zhannngchen @zxealous @zy-kkk @zzzxl1993 @zzzzzzzs