AMC Minor Technical Issues

Minor Technical Issues

Thread InitializationContinuation TrackingTime and std::chrono

Thread Initialization

Current thread architecture

Event threads and dedicated threads Event threads dispatch events from queues and do socket I/O. Dedicated threads execute a single event.

Threads are kept in a global table Each event thread has thread local storage which contains an Ethread

instance.

Thread Initialization

Thread initialization is done in a very unsafe manner. Threads are started. “Later”, but not too much later, the main thread updates data structures in

the other threads. Issues

Obviously unsafe, despite it seeming to work. Thread initialization is monolithic due to having to put all the logic in one

intervention call site to avoid more race conditions. Event loop update (TS-4260) removed an unused method that was used by

another patch (NUMA), need to add it back.

TS-4265Update thread initialization Change thread arrays to be arrays of structures to consolidate thread

information. Do thread initialization via continuations. Together these enable different parts of the process start logic to

execute code in another thread, during thread startup, without races with each other or the thread.

Intended for event threads. Not really needed for dedicated threads.

Start events

Implementation – re-use the “oneevent” member for event threads. Currently only used by dedicated threads. Change event threads to execute this event if set. Put a core event there that then executes a queue of other events.

Thread initialization means putting an event in this “spawn queue”.

Scheduling at spawn

EThread::schedule_spawn Single continuation per thread that is called first for regular thread.

EventProcessor::schedule_spawn Multiple functions that are invoked when an event thread is started. Built on top of thread schedule spawn.

Other changes

Event processor startup Net processor startup NUMA and processor affinity

Continuation Tracking

Track original source of continuations Originally designed for plugin priorities.

Need to know which plugin for a continuation to get the priority correct. Allows access to plugin registration and related data.

Requested as an independent feature. For debugging to track continuation problems back to specific plugins. Potentially useful for disabling plugins.

Implementation

Borrow a technique from per client IP debugging. Add per thread data item which points at the plugin registration data. Track current “plugin context”

During plugin load any continuations created are tagged with the plugin data.

During event dispatch Current plugin context is saved. Context is loaded from continuation. After event the old context is restored.

When a continuation is created it gets the current context.

Consequences

Plugin registration data pointer becomes continuation context. Need “core” plugin to represent continuations from the core and not a

plugin. Can go off track if plugins “trade” continuations. Dispatch can check plugin state before calling continuation handler.

Todo

Switch to C++ eleventy thread storage.

IP Allow Extension

IP Allow

Quinn’s project. Extend IP allow rules to control outbound connections based on the

origin IP address. Enable control of IP Allow rule enforcement in remap rules.

Currently if full DENY then connection closed before remap.

IP Allow implementation

Add “dest_ip” key to ip_allow.config. Add “ip_allow” as a remap filter so it can be turned off and on.

On by default Disable able early / accept check if any remap rule has IP Allow

disabled. Global effect –if any remap rule bypasses IP Allow then the fast check is

not done.

Example – allow outbound only to the 10/8 netdest_ip 10.0.0.0-10.255.255.255 action=ALLOWdest_ip 0.0.0.0-255.255.255.255 action=DENY

Example - remap

# ip_allow.configsrc_ip=127.0.0.1 action=ip_allow method=ALLsrc_ip=::1 action=ip_allow method=ALL

src_ip=0.0.0.0-255.255.255.255 action=ip_deny method=PUSH|PURGE|DELETEsrc_ip=::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff action=ip_deny method=PUSH|PURGE|DELETE# remap.config – disable ip_allow for the remaps, let everything else be denied.deactivatefilter ip_allowremap ….allow purge….remap … allow purge…..activatefilter ip_allow

std::chronoTime keeping in C++ eleventy

std::chrono and time

std::chrono is a library for handling time intervals. NOT calendar time! That’s a very different thing.

Instances are efficient Only the time unit is stored. Other properties are class properties known by the compiler. Conversions are known at compile time and inlined.

Conversions are semi-automatic. Conversions to finer resolution are automatic. Conversions to coarser resolution require explicit conversion. Information is never unintentionally lost.

std::chrono duration and time points

A duration is a length of time. A timepoint is a specific point in time.

Represented internally as a duration and an epoch. Epoch is a globally defined and known point in time.

E.g. standard UNIX epoch 1 Jan 1970.

std::chrono conversions and arguments Expressing exactly what time unit is desired is easy.

std::chrono::milliseconds(5) std::chrono::hours(1)

Automatic conversions let these be passed to any parameter that is at least as fined grained.

If methods take std::chrono::nanoseconds then passing time is easy. Caller does not need to know the parameter granularity unless it is coarser in which case it

is good to get a compile time error.

void leif_nap(std::chrono::nanoseconds length);

// ...

{ leif_nap(std::chrono::minutes(30));

leif_nap(std::chrono::days(2));

}

ATS example

// from P_CacheInternal.hint cache_config_mutex_retry_delay = 2;trigger = mutex->thread_holding->schedule_in_local(this, HRTIME_MSECONDS(cache_config_mutex_retry_delay), event);// vsts::milliseconds cache_config_mutex_retry_delay{2};trigger = mutex->thread_holding->schedule_in_local(this, cache_config_mutex_retry_delay, event);

std::chrono and clocks

std::chrono supports clocks system_clock monotonic_clock high_precision_clock My research indicates that in real life system_clock and

high_precision_clock are the same thing and sometimes monotonic_clock as well.

Time points are based on a specific clock because that clock embodies the epoch.

Durations are clock independent.

Active Projects

TS-974: Partial Object Caching TS-4260: Event loop improvements TS-4999: Plugin priority TS-4532: std::chrono TS-3347: make ink_assert a no-op in release mode. TS-4593: Extend IP allow to outbound connections.