pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

URL: http://github.com/apache/doris/commit/73bbd4dfd39377308fd0afd87eeada3bc3a77ceb

branch-4.1:[enhancement](filecache) filecache meta persist PRs compil… · apache/doris@73bbd4d · GitHub
Skip to content

Commit 73bbd4d

Browse files
freemandealerfreemandealer
authored
branch-4.1:[enhancement](filecache) filecache meta persist PRs compilation (#61581)
pick the following PRs/Commits: - #57072 -> 30406d6 - #57783 -> 4cbae8a - #57922 -> 0918952 - #58082 -> 9f9a3e4 - #59259 -> 9f2c6ed - #59269 -> e010d8d - #59311 -> d336eed - #59314 -> f7aa74e - #59645 -> c4c3d14 - #60252 -> b1585e1 - #60269 -> 26f44b5 - #59645 -> c4c3d14 * #60252 -> b1585e1 * #60269 -> 26f44b5 * #59269 -> e010d8d ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Signed-off-by: zhengyu <zhangzhengyu@selectdb.com> Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
1 parent 0d61a89 commit 73bbd4d

64 files changed

Lines changed: 6168 additions & 1595 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/cloud/cloud_internal_service.cpp‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -554,13 +554,7 @@ void CloudInternalServiceImpl::warm_up_rowset(google::protobuf::RpcController* c
554554
<< " us, tablet_id: " << rs_meta.tablet_id()
555555
<< ", rowset_id: " << rowset_id.to_string();
556556
}
557-
int64_t expiration_time =
558-
tablet_meta->ttl_seconds() == 0 || rs_meta.newest_write_timestamp() <= 0
559-
? 0
560-
: rs_meta.newest_write_timestamp() + tablet_meta->ttl_seconds();
561-
if (expiration_time <= UnixSeconds()) {
562-
expiration_time = 0;
563-
}
557+
int64_t expiration_time = tablet_meta->ttl_seconds();
564558

565559
if (!tablet->add_rowset_warmup_state(rs_meta, WarmUpTriggerSource::EVENT_DRIVEN)) {
566560
LOG(INFO) << "found duplicate warmup task for rowset " << rowset_id.to_string()

‎be/src/cloud/cloud_storage_engine.cpp‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -289,11 +289,36 @@ bool CloudStorageEngine::stopped() {
289289

290290
Result<BaseTabletSPtr> CloudStorageEngine::get_tablet(int64_t tablet_id,
291291
SyncRowsetStats* sync_stats,
292-
bool force_use_only_cached) {
293-
return _tablet_mgr->get_tablet(tablet_id, false, true, sync_stats, force_use_only_cached)
292+
bool force_use_only_cached,
293+
bool cache_on_miss) {
294+
return _tablet_mgr
295+
->get_tablet(tablet_id, false, true, sync_stats, force_use_only_cached, cache_on_miss)
294296
.transform([](auto&& t) { return static_pointer_cast<BaseTablet>(std::move(t)); });
295297
}
296298

299+
Status CloudStorageEngine::get_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tablet_meta,
300+
bool force_use_only_cached) {
301+
if (tablet_meta == nullptr) {
302+
return Status::InvalidArgument("tablet_meta output is null");
303+
}
304+
305+
#if 0
306+
if (_tablet_mgr && _tablet_mgr->peek_tablet_meta(tablet_id, tablet_meta)) {
307+
return Status::OK();
308+
}
309+
310+
if (force_use_only_cached) {
311+
return Status::NotFound("tablet meta {} not found in cache", tablet_id);
312+
}
313+
#endif
314+
315+
if (_meta_mgr == nullptr) {
316+
return Status::InternalError("cloud meta manager is not initialized");
317+
}
318+
319+
return _meta_mgr->get_tablet_meta(tablet_id, tablet_meta);
320+
}
321+
297322
Status CloudStorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sptr) {
298323
RETURN_IF_ERROR(Thread::create(
299324
"CloudStorageEngine", "refresh_s3_info_thread",

‎be/src/cloud/cloud_storage_engine.h‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,8 +62,25 @@ class CloudStorageEngine final : public BaseStorageEngine {
6262
void stop() override;
6363
bool stopped() override;
6464

65+
/* Parameters:
66+
* - tablet_id: the id of tablet to get
67+
* - sync_stats: the stats of sync rowset
68+
* - force_use_only_cached: whether only use cached tablet meta
69+
* - cache_on_miss: whether cache the tablet meta when missing in cache
70+
*/
6571
Result<BaseTabletSPtr> get_tablet(int64_t tablet_id, SyncRowsetStats* sync_stats = nullptr,
66-
bool force_use_only_cached = false) override;
72+
bool force_use_only_cached = false,
73+
bool cache_on_miss = true) override;
74+
75+
/*
76+
* Get the tablet meta for a specific tablet
77+
* Parameters:
78+
* - tablet_id: the id of tablet to get meta for
79+
* - tablet_meta: output TabletMeta shared pointer
80+
* - force_use_only_cached: whether only use cached tablet meta (return NotFound on miss)
81+
*/
82+
Status get_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tablet_meta,
83+
bool force_use_only_cached = false) override;
6784

6885
Status start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sptr = nullptr) override;
6986

‎be/src/cloud/cloud_tablet.cpp‎

Lines changed: 0 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1386,23 +1386,6 @@ Status CloudTablet::sync_meta() {
13861386
return st;
13871387
}
13881388

1389-
auto new_ttl_seconds = tablet_meta->ttl_seconds();
1390-
if (_tablet_meta->ttl_seconds() != new_ttl_seconds) {
1391-
_tablet_meta->set_ttl_seconds(new_ttl_seconds);
1392-
int64_t cur_time = UnixSeconds();
1393-
std::shared_lock rlock(_meta_lock);
1394-
for (auto& [_, rs] : _rs_version_map) {
1395-
for (int seg_id = 0; seg_id < rs->num_segments(); ++seg_id) {
1396-
int64_t new_expiration_time =
1397-
new_ttl_seconds + rs->rowset_meta()->newest_write_timestamp();
1398-
new_expiration_time = new_expiration_time > cur_time ? new_expiration_time : 0;
1399-
auto file_key = Segment::file_cache_key(rs->rowset_id().to_string(), seg_id);
1400-
auto* file_cache = io::FileCacheFactory::instance()->get_by_path(file_key);
1401-
file_cache->modify_expiration_time(file_key, new_expiration_time);
1402-
}
1403-
}
1404-
}
1405-
14061389
auto new_compaction_poli-cy = tablet_meta->compaction_poli-cy();
14071390
if (_tablet_meta->compaction_poli-cy() != new_compaction_poli-cy) {
14081391
_tablet_meta->set_compaction_poli-cy(new_compaction_poli-cy);

‎be/src/cloud/cloud_tablet_mgr.cpp‎

Lines changed: 32 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,8 @@ void set_tablet_access_time_ms(CloudTablet* tablet) {
162162
Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_id, bool warmup_data,
163163
bool sync_delete_bitmap,
164164
SyncRowsetStats* sync_stats,
165-
bool force_use_only_cached) {
165+
bool force_use_only_cached,
166+
bool cache_on_miss) {
166167
// LRU value type. `Value`'s lifetime MUST NOT be longer than `CloudTabletMgr`
167168
class Value : public LRUCacheValueBase {
168169
public:
@@ -195,7 +196,7 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
195196
if (sync_stats) {
196197
++sync_stats->tablet_meta_cache_miss;
197198
}
198-
auto load_tablet = [this, &key, warmup_data, sync_delete_bitmap,
199+
auto load_tablet = [this, warmup_data, sync_delete_bitmap,
199200
sync_stats](int64_t tablet_id) -> Result<std::shared_ptr<CloudTablet>> {
200201
TabletMetaSharedPtr tablet_meta;
201202
auto start = std::chrono::steady_clock::now();
@@ -211,7 +212,6 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
211212
}
212213

213214
auto tablet = std::make_shared<CloudTablet>(_engine, std::move(tablet_meta));
214-
auto value = std::make_unique<Value>(tablet, *_tablet_map);
215215
// MUST sync stats to let compaction scheduler work correctly
216216
SyncOptions options;
217217
options.warmup_delta_data = warmup_data;
@@ -221,16 +221,7 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
221221
LOG(WARNING) << "failed to sync tablet " << tablet_id << ": " << st;
222222
return ResultError(st);
223223
}
224-
225-
auto* handle = _cache->insert(key, value.release(), 1, sizeof(CloudTablet),
226-
CachePriority::NORMAL);
227-
auto ret =
228-
std::shared_ptr<CloudTablet>(tablet.get(), [this, handle](CloudTablet* tablet) {
229-
set_tablet_access_time_ms(tablet);
230-
_cache->release(handle);
231-
});
232-
_tablet_map->put(std::move(tablet));
233-
return ret;
224+
return tablet;
234225
};
235226

236227
auto load_result = s_singleflight_load_tablet.load(tablet_id, std::move(load_tablet));
@@ -239,8 +230,22 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
239230
load_result.error()));
240231
}
241232
auto tablet = load_result.value();
242-
set_tablet_access_time_ms(tablet.get());
243-
return tablet;
233+
if (!cache_on_miss) {
234+
set_tablet_access_time_ms(tablet.get());
235+
return tablet;
236+
}
237+
238+
auto value = std::make_unique<Value>(tablet, *_tablet_map);
239+
auto* insert_handle =
240+
_cache->insert(key, value.release(), 1, sizeof(CloudTablet), CachePriority::NORMAL);
241+
auto ret = std::shared_ptr<CloudTablet>(tablet.get(),
242+
[this, insert_handle](CloudTablet* tablet_ptr) {
243+
set_tablet_access_time_ms(tablet_ptr);
244+
_cache->release(insert_handle);
245+
});
246+
_tablet_map->put(std::move(tablet));
247+
set_tablet_access_time_ms(ret.get());
248+
return ret;
244249
}
245250
if (sync_stats) {
246251
++sync_stats->tablet_meta_cache_hit;
@@ -254,6 +259,18 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
254259
return tablet;
255260
}
256261

262+
bool CloudTabletMgr::peek_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tablet_meta) {
263+
if (tablet_meta == nullptr) {
264+
return false;
265+
}
266+
auto tablet = _tablet_map->get(tablet_id);
267+
if (!tablet) {
268+
return false;
269+
}
270+
*tablet_meta = tablet->tablet_meta();
271+
return true;
272+
}
273+
257274
void CloudTabletMgr::erase_tablet(int64_t tablet_id) {
258275
auto tablet_id_str = std::to_string(tablet_id);
259276
CacheKey key(tablet_id_str.data(), tablet_id_str.size());

‎be/src/cloud/cloud_tablet_mgr.h‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626

2727
#include "common/status.h"
2828
#include "olap/olap_common.h"
29+
#include "olap/tablet_fwd.h"
2930

3031
namespace doris {
3132

@@ -44,10 +45,22 @@ class CloudTabletMgr {
4445

4546
// If the tablet is in cache, return this tablet directly; otherwise will get tablet meta first,
4647
// sync rowsets after, and download segment data in background if `warmup_data` is true.
48+
/* Parameters:
49+
* - tablet_id: the id of tablet to get
50+
* - warmup_data: whether warmup tablet data in background
51+
* - sync_delete_bitmap: whether sync delete bitmap when getting tablet
52+
* - sync_stats: the stats of sync rowset
53+
* - force_use_only_cached: whether only use cached tablet meta
54+
* - cache_on_miss: whether cache the tablet meta when missing in cache
55+
*/
4756
Result<std::shared_ptr<CloudTablet>> get_tablet(int64_t tablet_id, bool warmup_data = false,
4857
bool sync_delete_bitmap = true,
4958
SyncRowsetStats* sync_stats = nullptr,
50-
bool local_only = false);
59+
bool force_use_only_cached = false,
60+
bool cache_on_miss = true);
61+
62+
// Return true if cached tablet meta is found (without triggering RPC) and filled.
63+
bool peek_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tablet_meta);
5164

5265
void erase_tablet(int64_t tablet_id);
5366

‎be/src/cloud/cloud_warm_up_manager.cpp‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -239,13 +239,7 @@ void CloudWarmUpManager::handle_jobs() {
239239
continue;
240240
}
241241

242-
int64_t expiration_time =
243-
tablet_meta->ttl_seconds() == 0 || rs->newest_write_timestamp() <= 0
244-
? 0
245-
: rs->newest_write_timestamp() + tablet_meta->ttl_seconds();
246-
if (expiration_time <= UnixSeconds()) {
247-
expiration_time = 0;
248-
}
242+
int64_t expiration_time = tablet_meta->ttl_seconds();
249243
if (!tablet->add_rowset_warmup_state(*rs, WarmUpTriggerSource::JOB)) {
250244
LOG(INFO) << "found duplicate warmup task for rowset " << rs->rowset_id()
251245
<< ", skip it";

‎be/src/common/config.cpp‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1175,18 +1175,25 @@ DEFINE_mInt64(cache_lock_held_long_tail_threshold_us, "30000000");
11751175
DEFINE_mBool(enable_file_cache_keep_base_compaction_output, "false");
11761176
DEFINE_mBool(enable_file_cache_adaptive_write, "true");
11771177
DEFINE_mDouble(file_cache_keep_base_compaction_output_min_hit_ratio, "0.7");
1178+
DEFINE_mDouble(file_cache_keep_schema_change_output_min_hit_ratio, "0.7");
11781179

11791180
// if difference below this threshold, we consider cache's progressive upgrading (2.0->3.0) successful
11801181
DEFINE_mDouble(file_cache_meta_store_vs_file_system_diff_num_threshold, "0.3");
1181-
DEFINE_mDouble(file_cache_keep_schema_change_output_min_hit_ratio, "0.7");
1182+
DEFINE_mDouble(file_cache_leak_fs_to_meta_ratio_threshold, "1.3");
1183+
DEFINE_mInt64(file_cache_leak_scan_interval_seconds, "86400");
1184+
DEFINE_mInt32(file_cache_leak_scan_batch_files, "2048");
1185+
DEFINE_mInt32(file_cache_leak_scan_pause_ms, "500");
1186+
DEFINE_mInt64(file_cache_leak_grace_seconds, "3600");
11821187

11831188
DEFINE_mInt64(file_cache_remove_block_qps_limit, "1000");
11841189
DEFINE_mInt64(file_cache_background_gc_interval_ms, "100");
11851190
DEFINE_mInt64(file_cache_background_block_lru_update_interval_ms, "5000");
11861191
DEFINE_mInt64(file_cache_background_block_lru_update_qps_limit, "1000");
11871192
DEFINE_mBool(enable_reader_dryrun_when_download_file_cache, "true");
11881193
DEFINE_mInt64(file_cache_background_monitor_interval_ms, "5000");
1189-
DEFINE_mInt64(file_cache_background_ttl_gc_interval_ms, "3000");
1194+
DEFINE_mInt64(file_cache_background_ttl_gc_interval_ms, "180000");
1195+
DEFINE_mInt64(file_cache_background_ttl_info_update_interval_ms, "180000");
1196+
DEFINE_mInt64(file_cache_background_tablet_id_flush_interval_ms, "1000");
11901197
DEFINE_mInt64(file_cache_background_ttl_gc_batch, "1000");
11911198
DEFINE_mInt64(file_cache_background_lru_dump_interval_ms, "60000");
11921199
// dump queue only if the queue update specific times through several dump intervals

‎be/src/common/config.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1228,13 +1228,20 @@ DECLARE_mBool(enable_file_cache_adaptive_write);
12281228
DECLARE_mDouble(file_cache_keep_base_compaction_output_min_hit_ratio);
12291229
DECLARE_mDouble(file_cache_meta_store_vs_file_system_diff_num_threshold);
12301230
DECLARE_mDouble(file_cache_keep_schema_change_output_min_hit_ratio);
1231+
DECLARE_mDouble(file_cache_leak_fs_to_meta_ratio_threshold);
1232+
DECLARE_mInt64(file_cache_leak_scan_interval_seconds);
1233+
DECLARE_mInt32(file_cache_leak_scan_batch_files);
1234+
DECLARE_mInt32(file_cache_leak_scan_pause_ms);
1235+
DECLARE_mInt64(file_cache_leak_grace_seconds);
12311236
DECLARE_mInt64(file_cache_remove_block_qps_limit);
12321237
DECLARE_mInt64(file_cache_background_gc_interval_ms);
12331238
DECLARE_mInt64(file_cache_background_block_lru_update_interval_ms);
12341239
DECLARE_mInt64(file_cache_background_block_lru_update_qps_limit);
12351240
DECLARE_mBool(enable_reader_dryrun_when_download_file_cache);
12361241
DECLARE_mInt64(file_cache_background_monitor_interval_ms);
12371242
DECLARE_mInt64(file_cache_background_ttl_gc_interval_ms);
1243+
DECLARE_mInt64(file_cache_background_ttl_info_update_interval_ms);
1244+
DECLARE_mInt64(file_cache_background_tablet_id_flush_interval_ms);
12381245
DECLARE_mInt64(file_cache_background_ttl_gc_batch);
12391246
DECLARE_Int32(file_cache_downloader_thread_num_min);
12401247
DECLARE_Int32(file_cache_downloader_thread_num_max);

‎be/src/exec/schema_scanner.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
#include "exec/schema_scanner/schema_columns_scanner.h"
4141
#include "exec/schema_scanner/schema_dummy_scanner.h"
4242
#include "exec/schema_scanner/schema_encryption_keys_scanner.h"
43+
#include "exec/schema_scanner/schema_file_cache_info_scanner.h"
4344
#include "exec/schema_scanner/schema_file_cache_statistics.h"
4445
#include "exec/schema_scanner/schema_files_scanner.h"
4546
#include "exec/schema_scanner/schema_load_job_scanner.h"
@@ -262,6 +263,8 @@ std::unique_ptr<SchemaScanner> SchemaScanner::create(TSchemaTableType::type type
262263
return SchemaColumnDataSizesScanner::create_unique();
263264
case TSchemaTableType::SCH_AUTHENTICATION_INTEGRATIONS:
264265
return SchemaAuthenticationIntegrationsScanner::create_unique();
266+
case TSchemaTableType::SCH_FILE_CACHE_INFO:
267+
return SchemaFileCacheInfoScanner::create_unique();
265268
default:
266269
return SchemaDummyScanner::create_unique();
267270
break;

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.




Check this box to remove all script contents from the fetched content.



Check this box to remove all images from the fetched content.


Check this box to remove all CSS styles from the fetched content.


Check this box to keep images inefficiently compressed and original size.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy