Ceph Cache Tiering and Objecter

  • 0 views

Based on Ceph v16.2.5

What does Objecter actually do?

librados-procedure
librados request procedure
osd-procedure
OSD I/O procedure

Objecter is located at Osdc.

Before sending a request to remote OSDService, it first need to determine which OSD, or rather PrimaryLogPG, it should talk to. Since the client side issue requests via the Ioctx context, which contains pg_pool_t, it already knows the read/write_tier of the base pool on which the Ioctx is initialized. So it does just that!
If issuing request to a pool configured with cache tier, librados always talks to the tier pool, only the PrimaryLogPG on the tier pool talks to the base pool.

The corresponding source code can be found at src/osdc/Objecter.cc/Objecter::_calc_target().

Fun fact, on the OSD side, if a cache operation is to be issued, it’s issued using an Objecter targeting base pool as well. OSD code referencing client side OSDC code, the later reference structs in OSD code, this is why Ceph code base is so fxxked up :joy:

Cache related OSD logic can be found at src/osd/PrimaryLogPG.cc/PrimaryLogPG::maybe_handle_cache_detail().

So where is the caching logic performed?

The caching logic is performed in the lengthy, one-function-do-it-all PrimaryLogPG::do_op(), which in turn calls PrimaryLogPG::maybe_handle_cache_detail().

PrimaryLogPG::maybe_handle_cache_detail(), with writeback cache mode, returns

  • NOOP
    1. on ignored
    2. on blocked
    3. on cache hit (I know, right?). After that, PrimaryLogPG::do_op() does the actual disk I/O.
  • HANDLED_PROXY
    1. on read cache miss, but cache is currently full. A PrimaryLogPG::do_proxy_read() is called and queued I/O before return.
    2. on read / write cache miss and room in cache. PrimaryLogPG::do_proxy_read/write() and PrimaryLogPG::maybe_promote(), i.e. PrimaryLogPG::promote_object() with some prepended logic, is called before return.
  • BLOCKED_FULL
    1. on write cache miss, but cache is currently full. A PrimaryLogPG::block_write_on_full_cache() is called before return.
  • BLOCKED_PROMOTE
    1. on cache miss and explicit promote
    2. concurrent promote