Double Trouble: Fuzzing Kernel Modules Derrick McKee, Prashast Srivastava, Hui Peng, Mathias Payer Abstract Despite its importance, the Linux kernel remains a difficult target for fuzzers. While prior work, such as syzkaller, allows for fuzzing the kernel, it is still difficult to fuzz arbitrary loadable kernel modules (LKMs). These LKMs represent the majority of source code in the Linux kernel, as well as a major source of CVEs. We present Double Trouble, a general framework for fuzzing LKMs. Double Trouble requires a one-time minimal programming effort to support fuzzing any subsystem. Once that effort is complete, any LKM within the subsystem can be fuzzed from the two possible sources of inputs: an upper half direction representing a system call execution, and a lower half direction representing either hardware interrupt or exposed kernel function. As a case study, we are adding support for fuzzing filesystems. We use the state-of-the-art generational fuzzer, af-fuzz, for the upper half fuzzing, and a deterministic fuzzer for the lower half. Problems ● 568 critical vulnerabilities found in the Linux kernel last year. ● Emulating a whole system drastically limits fuzzing throughput. ● The complexity of the kernel also greatly hinders the fuzzer code coverage. ● Often difficult to fuzz specific modules. mutex_lock _cond_resched msdos_format_name strchr fat_scan strncmp fat_get_short_entry __brelse __warn_printk mutex_unlock current_time current_kernel_time64 fat_alloc_new_dir fat_free_clusters fat_alloc_clusters __getblk_gfp fat_time_unix2fat __memset mark_buffer_dirty_inode fat_zeroed_cluster.constprop.25 msdos_add_entry fat_add_entries fat_sync_inode __mark_inode_dirty inc_nlink fat_build_inode fat_iget new_inode iunique fat_fill_inode iput fat_attach __insert_inode_hash set_nlink d_instantiate security_d_instantiate _raw_spin_lock __d_instantiate fat_flush_inodes writeback_inode fat_collect_bhs mark_fsinfo_dirty fat_mirror_bhs fat_sync_bhs fat_ent_read blkdev_issue_discard __fat_fs_error msdos_mkdir d_flags_for_inode printk vprintk fat__get_entry inode_needs_sync __find_get_block pagecache_get_page try_to_free_buffers alloc_page_buffers init_page_buffers unlock_page __put_page dump_stack time64_to_tm mark_buffer_dirty __bforget fat_msg __memcpy sync_dirty_buffer __fat_remove_entries fat_add_new_entries fat_chain_add block_dump___mark_inode_dirty inode_io_list_move_locked I_BDEV write_dirty_buffer __wait_on_buffer sync_inode_metadata __bread_gfp ___ratelimit fat_bmap __breadahead page_mapping __set_page_dirty writeback_single_inode out_of_line_wait_on_bit_lock rcu_all_qs __schedule __wait_on_bit_lock d_find_alias wb_io_lists_depopulated blk_start_plug __blkdev_issue_discard submit_bio_wait bio_put blk_finish_plug panic mark_page_accessed find_get_entry __page_cache_alloc add_to_page_cache_lru __lock_page drop_buffers free_buffer_head __cancel_dirty_page alloc_buffer_head set_bh_page.part.29 __page_cache_release vprintk_func dump_stack_print_info show_stack __inode_wait_for_writeback mapping_tagged __writeback_single_inode inode_to_wb_and_lock_list inode_sync_complete inode_io_list_del_locked submit_bh_wbc.isra.46 unlock_buffer out_of_line_wait_on_bit _raw_spin_trylock _raw_spin_unlock_irqrestore printk_deferred ll_rw_block rcu_note_context_switch switch_mm_irqs_off __switch_to_asm finish_task_switch deactivate_task update_rq_clock.part.80 ___perf_sw_event enter_lazy_tlb __perf_event_task_sched_out __delayacct_blkio_start wq_worker_sleeping __schedule_bug profile_hits __balance_callback activate_task wq_worker_waking_up ttwu_do_wakeup.isra.82 ttwu_stat.part.49 __delayacct_blkio_end vprintk_deferred igrab new_inode_pseudo inode_sb_list_add get_seconds fat_calc_dir_size fat_subdirs fat_time_fat2unix _atomic_dec_and_lock list_lru_del write_inode_now inode_lru_list_add rcu_sched_qs rcu_dynticks_momentary_idle load_new_mm_cr3 rcu_irq_enter_disabled rcu_irq_enter_irqson rcu_irq_exit_irqson __pti_set_user_pgd clear_asid_other __perf_event_task_sched_in __mmdrop kprobe_flush_task put_task_stack __put_task_struct sched_clock_cpu perf_exclude_event perf_swevent_event perf_pmu_sched_task perf_iterate_sb task_ctx_sched_out perf_event_update_time perf_event_update_userpage ktime_get kthread_data print_modules _raw_spin_lock_irqsave check_preempt_curr __const_udelay panic_smp_self_stop bust_spinlocks vsnprintf printk_safe_flush_on_panic __crash_kexec crash_smp_send_stop atomic_notifier_call_chain kmsg_dump debug_locks_off console_flush_on_panic emergency_restart __mod_node_page_state __mod_zone_page_state vprintk_emit irq_work_queue fat_get_cluster fat_ent_write fat_cache_inval_inode strcspn print_tainted queued_spin_lock_slowpath resched_curr activate_page fat_cache_add.part.10 kmem_cache_free list_lru_add __printk_safe_enter vscnprintf cont_flush log_store __printk_safe_exit cont_add __down_trylock_console_sem.isra.20 console_trylock.part.21 console_unlock llist_add_batch arch_irq_work_raise tick_nohz_tick_stopped native_sched_clock sched_clock_local recalc_bh_state.part.44 perf_iterate_ctx calc_timer_values arch_perf_update_userpage finish_wait prepare_to_wait_exclusive bio_alloc_bioset bio_add_page guard_bio_eod submit_bio __wait_on_bit mempool_alloc punt_bios_to_rescuer bvec_alloc __kmalloc mempool_free __disk_get_part generic_make_request __bdevname this_cpu_cmpxchg16b_emu __slab_free cpumask_next prepare_to_wait kmem_cache_alloc fat_cache_merge cmpxchg_double_slab.isra.73 free_debug_processing using_native_sched_clock sched_clock_stable cyc2ns_read_begin cyc2ns_read_end remove_element.isra.9.part.10 io_schedule_timeout kmalloc_slab __slab_alloc log_make_free_space down_trylock __up_console_sem.isra.19 scnprintf msg_print_text sprintf msg_print_ext_body ___slab_alloc generic_make_request_checks blk_queue_enter bio_endio account_page_dirtied radix_tree_tag_set __page_file_index pagevec_lru_move_fn alloc_inode release_pages module_flags up bit_waitqueue do_writepages is_bad_inode filemap_fdatawait_range inode_add_lru init_object on_freelist check_object set_track print_hex_dump slab_err slab_fix slab_bug print_trailer check_slab unblank_screen console_unblank format_decode number pointer string set_precision set_field_width __sw_hweight64 mutex_trylock crash_setup_regs crash_save_vmcoreinfo machine_crash_shutdown machine_kexec console_trylock __cmpxchg_double_slab.isra.61 get_partial_node.isra.84 gfp_pfmemalloc_allowed deactivate_slab.isra.82 mempolicy_slab_node __cpuset_node_allowed __next_zones_zonelist new_slab slab_out_of_memory alloc_debug_processing next_bio wait_for_completion_io blk_status_to_errno blk_flush_plug_list check_bytes_and_report save_stack_trace hex_dump_to_buffer add_taint print_tracking.part.58 slab_pad_check.part.64 rcu_irq_enter rcu_irq_exit print_track vmcoreinfo_append_str relocate_kernel disable_IO_APIC radix_tree_lookup_slot __alloc_pages_nodemask cpuset_mem_spread_node __add_to_page_cache_locked lru_cache_add workingset_refault workingset_activation _raw_spin_lock_irq io_schedule ctx_sched_out perf_event_sched_in.isra.117 pgd_free destroy_context_ldt __mmu_notifier_mm_destroy kfree recycle_rp_inst account_kernel_stack arch_release_thread_stack vfree_atomic __free_pages cgroup_free security_task_free exit_creds profile_handoff_task queue_work_on module_flags_taint memchr_inv __put_compound_page free_unref_page_list find_next_bit __radix_tree_lookup bio_chain free_ldt_struct.part.6 put_cpu_partial discard_slab __next_node_in init_tracking.part.68 mod_node_page_state alloc_pages_current count_partial __up.isra.2 group_sched_out.part.103 ctx_sched_in.isra.115 clear_IO_APIC __inc_zone_page_state __inc_node_page_state percpu_counter_add_batch mod_zone_page_state console_lock blk_mq_flush_plug_list list_sort queue_unplugged __elv_add_request __blk_end_request_all PageHuge free_unref_page_prepare free_unref_page_commit.isra.92 congestion_wait write_cache_pages __filemap_fdatawait_range pagevec_lookup_range_tag wait_on_page_bit_common __pagevec_release event_sched_out.isra.100 unfreeze_partials.isra.83 find_get_pages_range_tag lru_add_drain_cpu handle_bad_sector create_task_io_context init_wait_entry schedule prepare_to_wait_event io_schedule_prepare schedule_timeout free_pages inode_init_always down security_inode_alloc __init_rwsem skip_atoi put_dec dentry_name pointer_string siphash_1u64 restricted_pointer uuid_string resource_string device_node_string.isra.5 netdev_bits mac_address_string escaped_string clock ip4_addr_string bitmap_list_string.isra.6 address_val symbol_string hex_string bdev_name.isra.8 flags_string bitmap_string.isra.7 ip6_addr_string_sa ip6_addr_string ip4_addr_string_sa lock_timer_base detach_if_pending __internal_add_timer trigger_dyntick_cpu.isra.35 del_timer_sync get_nohz_timer_target get_page_from_freelist __alloc_pages_slowpath policy_nodemask policy_node get_task_policy.part.41 jiffies_to_usecs wait_on_page_bit clear_page_dirty_for_io tag_pages_for_writeback print_prefix memchr snprintf node_random group_sched_in perf_event_set_state.part.76 radix_tree_maybe_preload page_cache_tree_insert lruvec_lru_size find_first_bit refcount_dec_not_one put_css_set_locked __put_cred __queue_work clear_IO_APIC_pin pagevec_move_tail has_capability_noaudit string_escape_mem __clk_get_name clk_get_rate ip4_string sprint_symbol sprint_symbol_no_offset sprint_backtrace strlen ip6_string ip6_compressed_string __down kmem_cache_alloc_node radix_tree_next_chunk blk_mq_sched_insert_requests __blk_run_queue blk_run_queue_async elv_drain_elevator elv_attempt_insert_merge elv_rqhash_add __pm_runtime_resume __blk_end_bidi_request ioapic_read_entry __eoi_ioapic_pin ioapic_mask_entry ioapic_write_entry cpuset_nodemask_valid_mems_allowed free_pcppages_bulk __radix_tree_create shmem_mapping __radix_tree_replace security_capable_noaudit event_sched_in.isra.113.part.114 perf_mux_hrtimer_restart radix_tree_node_alloc.constprop.25 radix_tree_extend node_tag_get.constprop.24 __zone_watermark_ok node_dirty_ok __inc_numa_state clear_page_orig get_pfnblock_flags_mask set_pageblock_migratetype move_freepages_block __node_distance node_reclaim find_suitable_fallback steal_suitable_fallback check_new_page_bad __cpuset_memory_pressure_bump try_to_free_pages unreserve_highatomic_pageblock drain_all_pages __alloc_pages_direct_compact zone_reclaimable_pages schedule_timeout_uninterruptible wake_all_kswapds out_of_memory compaction_zonelist_suitable warn_alloc put_dec_full8 get_random_u32 perf_log_itrace_start perf_log_throttle hrtimer_forward hrtimer_start_range_ns free_pages_check_bad page_mkclean dec_zone_page_state set_page_dirty radix_tree_iter_tag_set radix_tree_iter_resume fmeter_update throttle_direct_reclaim do_try_to_free_pages first_online_pgdat next_zone flush_work try_to_compact_pages compaction_defer_reset wakeup_kswapd task_will_free_mem oom_kill_process schedule_timeout_killable blocking_notifier_call_chain mark_oom_victim wake_oom_reaper.part.28 has_intersects_mems_allowed dump_header.isra.30 oom_badness __compaction_suitable.part.38 cpuset_print_current_mems_allowed show_mem get_work_pool check_flush_dependency __init_waitqueue_head insert_work wait_for_completion node_page_state node_pagecache_reclaimable shrink_node blk_attempt_req_merge elv_rqhash_find rpm_resume blk_mq_insert_requests blk_update_bidi_request blk_finish_request blk_update_request add_disk_randomness blk_delete_timer blk_account_io_done blk_queue_end_tag blk_stat_add laptop_io_completion __blk_put_request allow_direct_reclaim.part.78 compaction_suitable __delayacct_freepages_start zone_watermark_ok_safe __delayacct_freepages_end blk_account_io_completion bio_advance blk_dump_rq_flags blk_recalc_rq_segments add_timer_randomness __ioapic_read_entry part_round_stats part_dec_in_flight elv_completed_request freed_request elv_put_request put_io_context __wake_up __pm_runtime_idle del_timer __rpm_get_callback dev_pm_disable_wake_irq_check rpm_callback dev_pm_enable_wake_irq_check rpm_idle _warn_unseeded_randomness _raw_read_lock_irqsave extract_crng _raw_read_unlock_irqrestore pr_cont_kernfs_name show_free_areas next_online_pgdat page_mapped rmap_walk pgdat_balanced shrink_node_memcg.constprop.84 shrink_slab.part.69 part_in_flight queued_read_lock_slowpath sum_zone_node_page_state inactive_list_is_low.constprop.87 shrink_active_list shrink_inactive_list down_read_trylock up_read fragmentation_index kernfs_name_locked __freed_request attempt_merge lock_hrtimer_base.isra.20 enqueue_hrtimer __remove_hrtimer tick_program_event wake_up_nohz_cpu ktime_get_seconds mix_pool_bytes credit_entropy_bits show_mem_node_skip.part.76 hugetlb_show_meminfo show_swap_cache_info call_rwsem_wake zone_watermark_ok strlcpy calc_wheel_index try_to_del_timer_sync housekeeping_test_cpu compaction_deferred compact_zone_order defer_compaction put_dec_trunc8 perf_output_begin __perf_event_header__init_id perf_output_copy __perf_event__output_id_sample perf_output_end perf_event_pid_type cpumask_next_and __blk_recalc_rq_segments find_lock_task_mm _raw_read_lock put_task_struct do_send_sig_info down_read notifier_call_chain __thaw_task mempolicy_nodemask_intersects cpuset_mems_allowed_intersects dump_unreclaimable_slab int_sqrt lru_add_drain isolate_lru_pages.isra.79 page_evictable page_referenced move_active_pages_to_lru putback_lru_page try_to_release_page msleep shrink_page_list putback_inactive_pages current_may_throttle wakeup_flusher_threads wait_iff_congested blk_mq_hctx_mark_pending.isra.43 get_slabinfo blk_rq_set_mixed_merge elv_merge_requests ioprio_best rwsem_wake perf_event_header__init_id perf_event__output_id_sample perf_output_put_handle __rpm_callback rpm_check_suspend_allowed total_swapcache_pages clear_wb_congested __msecs_to_jiffies __isolate_lru_page try_to_free_swap try_to_unmap_flush add_to_swap __remove_mapping try_to_unmap try_to_unmap_flush_dirty inc_node_page_state __filemap_set_wb_err __wakeup_flusher_threads_bdi.part.53 __fragmentation_index timerqueue_add timerqueue_del clockevents_switch_state __page_mapcount page_rmapping add_page_to_unevictable_list isolate_lru_page __lock_task_sighand send_signal __dev_pm_qos_read_value device_links_read_lock device_links_read_unlock rb_insert_color wb_wakeup __task_pid_nr_ns elv_rqhash_reposition elv_rqhash_del.part.12 _raw_spin_lock_bh mod_delayed_work_on __rwsem_mark_wake.constprop.5 wake_up_q call_rwsem_down_read_failed wake_up_process page_swapcount pm_suspended_storage delete_from_swap_cache arch_tlbbatch_flush get_swap_page add_to_swap_cache put_swap_page __delete_from_swap_cache __delete_from_page_cache workingset_eviction rmap_walk_locked errseq_set try_to_grab_pending __queue_delayed_work task_active_pid_ns rb_erase rb_next cpumask_any_but native_flush_tlb_others flush_tlb_func_common.constprop.11 rwsem_down_read_failed _swap_info_get try_to_wake_up get_swap_pages __drain_swap_slots_cache.constprop.1 radix_tree_maybe_preload_order __add_to_swap_cache __swap_entry_free __radix_tree_insert radix_tree_delete plist_requeue __del_from_avail_list scan_swap_map_slots drain_slots_cache_cpu.constprop.2 swapcache_free_entries kvfree unaccount_page_cache_page radix_tree_clear_tags __swap_info_get swap_count_continued pwq_dec_nr_in_flight pwq_activate_delayed_work set_task_cpu __task_rq_lock ttwu_do_activate.isra.83 select_fallback_rq dump_page node_tag_clear set_task_rq_fair __printk_ratelimit do_set_cpus_allowed cpuset_cpus_allowed_fallback plist_del scan_swap_map_try_ssd_cluster inc_cluster_info_page compact_zone _mix_pool_bytes kill_fasync crng_reseed swap_do_scheduled_discard __free_cluster migrate_prep_local isolate_migratepages_block drain_local_pages putback_movable_pages __reset_isolation_suitable __pageblock_pfn_to_page migrate_pages release_freepages __update_load_avg_blocked_se.isra.33 vmalloc_to_page complete sort add_to_avail_list send_sigio extract_entropy.constprop.43 memzero_explicit invalidate_batched_entropy module_put _extract_crng _crng_backtrack_protect compact_unlock_should_abort.isra.35 compact_trylock_irqsave.isra.20 update_pageblock_skip isolate_movable_page putback_active_hugepage PageMovable putback_movable_page set_pfnblock_flags_mask size_to_hstate move_to_new_page __put_anon_vma page_get_anon_vma remove_migration_ptes xfer_secondary_pool account.constprop.44 _raw_write_lock_irqsave chacha20_block plist_add __wake_up_locked queued_write_lock_slowpath decay_load send_sigio_to_task migrate_page down_write up_write security_file_send_sigiotask migrate_page_move_mapping migrate_page_copy migrate_page_states call_rwsem_down_write_failed rwsem_down_write_failed radix_tree_replace_slot __dec_node_state __inc_node_state buffer_migrate_lock_buffers __dec_zone_state __inc_zone_state copy_page __lock_buffer osq_lock rwsem_spin_on_owner osq_unlock __rwsem_mark_wake.part.2 wake_q_add Our Solution ● Ignore kernel features that are orthogonal to correct module functionality. ● Implement “kernel gateway functions” that provide simple services, such as memory allocation. ● Provide a generic, simple API that developers can use to exercise their module functionality. ● Generate a user space application that is as fast and scalable as the underlying fuzzer. mutex_lock fat_dir_empty fat_get_short_entry __brelse mutex_unlock msdos_find msdos_format_name fat_scan fat_remove_entries current_time mark_buffer_dirty_inode sync_dirty_buffer fat_sync_inode __fat_remove_entries fat_msg __mark_inode_dirty drop_nlink clear_nlink fat_detach _raw_spin_lock fat_flush_inodes writeback_inode msdos_rmdir fat__get_entry printk __bread_gfp ___ratelimit fat_bmap __find_get_block __breadahead strchr strncmp