Search Jobs
EMBEDDED OS ENGINEER – Linux Firmware
All, Software/SaaS
Salt Lake City, UT
Location: Salt Lake City, UT | Schedule: Full-time, on-site
Overview
An advanced hardware technology company building an autonomous drone platform powered by the Qualcomm QRB5165 processor is seeking a hands-on Embedded OS Engineer. This person will own the Ubuntu-based Linux firmware stack for the platform, ensuring it is secure, reliable, and shipped through automated CI/CD pipelines. The role requires deep comfort with low-level OS internals rather than application-layer development.
What you'll do
- Own and maintain the build repository, including layer configuration, local.conf tuning, and image recipes used to produce the production firmware image
- Write and maintain .bb recipes and .bbappend overlays; create .patch files (git diff-based) against the pinned Qualcomm QRB5165 BSP (LU.UM.3.3.1) so upstream sources are never modified directly
- Debug BitBake task failures across the do_fetch to do_compile to do_rootfs to do_image pipeline, understand sstate-cache behavior, and know when to use -c cleansstate vs. -c rootfs -f
- Optimize build performance by tuning BB_NUMBER_THREADS and PARALLEL_MAKE, understanding QEMU emulation overhead vs. native ARM64 compilation, and leveraging shared sstate-cache across Jenkins agents
- Manage package inclusion and exclusion (IMAGE_INSTALL:remove, PACKAGE_EXCLUDE, BAD_RECOMMENDATIONS) and inter-recipe dependency graphs
- Implement ROOTFS_POSTPROCESS_COMMAND functions for postbuild OS customization, including package upgrades, apt security patching (Ubuntu ESM), service installation, and file system fixups
- Remove unnecessary services from production images (HTTP servers such as lighttpd, TFTP daemons, and other attack-surface-expanding services) identified via Blue List / Nessus security assessments
- Write bash scripts and systemd service units that run reliably in a headless, rootfs post-install context with no interactive terminal and limited /proc and /dev availability
- Manage the dpkg / apt ecosystem within the embedded rootfs, including package pinning, held packages, validating package state, and keeping apt lock files clean across incremental builds
- Maintain and improve Jenkins pipelines that build Yocto firmware images, including concurrent build isolation, artifact staging, and race-condition handling between parallel jobs
- Diagnose and resolve Jenkins agent performance issues including Java heap tuning, Docker overlay2 disk pressure, container lifecycle management, and build environment reproducibility
- Implement proper artifact copy patterns (stageDir isolation) to prevent race conditions when multiple builds run concurrently against shared directories
- Manage GitLab repository structure and branch protection for the build repo, implementing .patch-based change workflows that preserve the integrity of pinned upstream BSP branches
- Configure systemd-networkd for runtime network mode switching (DHCP to link-local) on headless devices using udev rules, button-press event handlers, and LED feedback mechanisms
- Write and maintain systemd service units and timers, understanding WantedBy, After, Requires, and ExecStart semantics for embedded boot sequences
- Implement reliable USB logging and file transfer services (usb-gadget, udevadm) for field diagnostics on devices with no screen
- Develop and maintain Prometheus node_exporter integrations and process-exporter configurations for fleet health monitoring via Grafana dashboards
- Remediate Nessus / Blue List security findings affecting the embedded OS, including SSH hardening (key-only auth, sshd_config tuning), open port reduction, and service inventory documentation
- Maintain /etc/shadow hygiene, locked account policies, and PAM configurations appropriate for production embedded devices
- Generate and maintain security compliance artifacts (port inventories, service lists, patch status reports) for internal security assessors
- Profile and diagnose slow boot sequences, runaway processes, and memory pressure on ARM64 embedded hardware
- Use screen, tmux, and remote shell tooling to manage long-running build and deployment sessions on headless servers and devices
- Implement OS-level monitoring including log capture services, boot-time diagnostics, and watchdog patterns for unattended field deployment
- Advise on cloud build infrastructure choices (AWS Graviton / Azure ARM64) to eliminate QEMU emulation overhead and achieve 3-5x build time reductions
Required qualifications
- Hands-on experience with Yocto and BitBake build systems, including layer configuration and recipe/overlay development
- Experience working with Qualcomm QRB5165 or similar embedded BSPs
- Strong Linux internals knowledge beyond the application layer, including systemd, networking, and package management (dpkg/apt)
- Experience building and maintaining Jenkins CI/CD pipelines for firmware or embedded builds
- Experience with GitLab repository management and patch-based workflows
- Proficiency writing bash scripts and systemd service units for headless embedded environments
- Experience with OS security hardening, including SSH configuration, PAM, and remediation of Nessus/security scan findings
- Experience with Docker and container lifecycle management in a build environment
- Experience with ARM64 architecture and build optimization, including understanding of QEMU emulation overhead
- Familiarity with Prometheus and Grafana for system monitoring
Why this role
This is a high-ownership position building and securing the core OS platform for an autonomous drone system, combining low-level embedded Linux engineering with full CI/CD pipeline ownership.
"*" indicates required fields