README.md

# lenix

A WIP POSIX-compliant microkernel and Unix-like operating system

## Vision

Lenix targets a secure, efficient, enterprise-grade footprint that can scale from single-board computers to virtualized deployments (QEMU/KVM, OpenBSD VMM, and Bhyve). The initial bring-up focuses on x86_64 and arm64/aarch64 with planned RISC-V64 support once the core subsystems mature. Symmetric multiprocessing and distributed capabilities are in scope from the outset so the system can span multi-core boards and clustered environments.

The kernel follows a microkernel architecture inspired by Minix, GNU/Hurd, Mach,as well as OpenBSD, IRIX, OpenVMS, and Plan 9 - emphasizing clear privilege boundaries, message-passing services, and network-transparent interfaces.

Because IPC is the lifeblood of any microkernel, Lenix will grow a first-class message-passing framework that lets user-mode servers exchange capabilities, share memory safely, and marshal POSIX-style requests without collapsing isolation guarantees. Every subsystem we add (drivers, filesystems, network stacks) is expected to live behind that IPC surface so scaling out to distributed deployments remains straightforward.

We will have real asynchronous I/O supported directly by the kernel. We also want a comprehensive, fine-grained security model involving a database of subjects, actions, and objects, giving administrators fine control over who may do what to what. In short, ACLs built into the OS from the ground up. We want to also automatically log all events, providing an audit trail for security incidents. We should also support clustering to complement IPC; support for failover and load balancing should be built into the OS.

The operating system should never crash (Hah!).

Standards compliance is a first-class goal: the project aims to conform to the Single UNIX Specification UNIXV7 - POSIX.1-2017 (IEEE Std 1003.1-2017) minimum as capabilities come online, looking towards POSIX.1-2024 expansion.

We will begin with tarfs+tmpfs support, then ext2, then UFS2. ZFS would be desired as the project matures.

We will use musl or another modern, POSIX-complient libc implementation (or, preferably, our own implementation).

Some of the following documentation may be outdated - check changelog.md and roadmap.md for a better idea of the current state of the microkernel.

## Known major issues and blockers

- IPC/VFS timing (or something else!) can cause intermittent boot problems.
- Intermittent hangs
- arm64 support is lagging behind
- SMP needs work

## Toolchain

Current status: we build an LLVM/clang cross-compiler in `/opt/llvm-mercuron` targeting `x86_64-unknown-mercuron`. See `third-party/ports/llvm`.

- **Clang/LLVM:** 21.1.5 (see `docs/toolchain.md`)

The `Makefile` assumes the `clang` binary on your PATH is version 21.1.5:

```bash
$ clang --version
clang version 21.1.5
```

If the version differs, install LLVM 21, export `CC=/path/to/clang-21`, and rebuild.

## Dependencies

Fully tested on Arch Linux

### Arch Linux

````
sudo pacman -Syu --needed base-devel git \
  clang lld lldb llvm \
  qemu-system-x86 qemu-system-aarch64 gdb nasm \
  xorriso mtools dosfstools grub ovmf \
  cmake ninja meson bear ccache python
````

### Debian

````
sudo pacman -Syu --needed base-devel git \
  clang lld lldb llvm \
  qemu-system-x86 qemu-system-aarch64 gdb nasm \
  xorriso mtools dosfstools grub ovmf \
  cmake ninja meson bear ccache python
````

### OpenBSD

````
doas pkg_add gmake nasm qemu ccache cmake ninja meson pkgconf \
             python3 clang-tools-extra cppcheck
````

- VMM firmware: vmm-firmware

### FreeBSD

````
sudo pkg install -y gmake nasm qemu-nox11 ccache cmake ninja meson pkgconf \
                    python llvm-devel clang-tidy cppcheck uefi-edk2-bhyve
````

- Bhyve firmware: uefi-edk2-bhye

## Building

`./rebuild-debug.sh` or `./rebuild.sh`

Rebuild images with musl staged: `gmake rootfs` and/or `gmake initrd`. Set PACKAGE_MUSL=0 to skip entirely or INCLUDE_MUSL_HEADERS=0 to drop headers while keeping the loader/libs.


`ARCH` selects the target architecture (default `x86_64`). For example:

```bash
# x86_64 legacy + UEFI artifacts:
make clean && make ARCH=x86_64 -j4 && make package-runtime && make package-apps && make initrd && make efi && make iso

# ARM64 (QEMU virt) legacy kernel:
ARCH=arm64 make legacy
```
## Booting

**UEFI (default)**
```bash
make run-uefi
```

**Legacy (Multiboot2/GRUB)**

```
make run-legacy

# or:

qemu-system-x86_64 -cdrom build/x86_64/lenix.iso -m 2048M -smp 2 -serial stdio -display none  -drive if=none,id=virtio-rootfs,file="build/rootfs.ext2",format=raw -device virtio-blk-pci,drive=virtio-rootfs
````

**ARM64 (QEMU virt)**

````
ARCH=arm64 make run-arm64  # spawns a 2-core virt machine
````

## Testing

A cross-arch smoke test is available to ensure the kernel, scheduler, syscall
path, and IPC scaffolding still boot to the expected point. It builds both
architectures, runs them under QEMU, and now asserts that the scheduler reaches
the fs-server launch point *and* that the telemetry task logs a heartbeat (so we
know timer ticks are flowing and user-mode threads are alive) - needs updating:

````
./scripts/test_sanity.sh
make lint              # clang-tidy/cppcheck/style gate
````

Set `SANITY_TIMEOUT=<seconds>` to adjust how long each QEMU run is allowed to
execute before the harness kills it (default: 25s).

> **Note:** ARM64 bring-up targets QEMU's `virt` machine. Install `qemu-system-aarch64`
> (or ensure your QEMU package includes ARM64 targets) so `ARCH=arm64` builds can run locally.

## Initrd packaging

Build the initial initrd bundle (cpio/newc) with the current user-mode servers:

````
make initrd        # writes build/initrd.cpio
````

Set PACKAGE_MUSL=0 to skip musl packaging entirely or INCLUDE_MUSL_HEADERS=0 to drop headers while keeping the loader/libs.

Contents include init, namesvc, vfs, fs.ext2, blockd, ramdiskd, virtio-blk, pci, ttyd, ptyctl, and demo.

The UEFI loader will automatically map `EFI/BOOT/INITRD.CPIO` if present. Running `make esp` or `make run-uefi` copies `build/initrd.cpio` into place; the legacy/GRUB ISO now carries `/boot/initrd.cpio` (see `boot/grub/grub.cfg`). Kernel user daemons prefer binaries from the initrd and fall back to the embedded images if missing.




## SMP status

### SMP CURRENTLY DISABLED BY DEFAULT 
Lenix now boots as a true multi-core kernel on both supported architectures. The
x86_64 path enumerates CPUs through legacy MP tables when available and falls
back to ACPI MADT parsing (via the UEFI system table or a BIOS scan) so modern
machines no longer strand secondary cores. arm64 keeps using PSCI `CPU_ON`
requests but now wires SGIs for both reschedule and generic IPIs, ensuring the
per-CPU scheduler queues can be nudged from any core. The shared scheduler owns
one run queue per CPU, automatically balances runnable tasks across them, and
requests remote reschedules via SMP IPIs whenever userland enqueues work on a
different core.

To keep IPC reliable under contention, mailboxes now guard their head/tail
pointers with lightweight spin locks, so concurrent portal deliveries from
multiple CPUs no longer corrupt queues or trip the diagnostic watchdog. If SMP
firmware data is missing, the kernel logs the condition and continues in
single-core mode rather than panicking.

## IPC scaffolding

Every scheduler task now owns a per-task mailbox (`kernel/ipc/mailbox.c`) so kernel services and staged user processes can pass small control messages without sharing stacks. The scheduler allocates these queues during task creation, exposes them through `sched_task_mailbox()`, and drains them on teardown so messages never leak across processes. See `docs/ipc.md` for an overview of the mailbox lifecycle and how it fits into the Minix/Hurd-inspired IPC roadmap.

Capability-style portals (`kernel/ipc/portal.c`) sit on top of those mailboxes. Kernel services mint handles that point at a target task’s mailbox plus a rights mask, then hand the handle to trusted clients. Calls such as `ipc_portal_send()` validate the capability before copying data into the destination queue, laying the groundwork for user-visible IPC APIs and distributed message routers.

Portals can also record a preferred CPU so NUMA-aware services (filesystems, network stacks, etc.) can keep requests close to the cores that host their state. When a hint is present the scheduler nudges that CPU via an SMP IPI so work migrates toward the desired locality without busy waiting. Mailboxes/portals also expose remote placeholders (node IDs + transport stubs) so future distributed transports can route a message over the network instead of delivering it locally.

User-mode tasks now resolve services via the nameserver: `service_resolve("vfs"/"fs.ext2"/"blockd"/"tty.console")` talks to namesvc (distributed via `SYS_service_portal(LENIX_SERVICE_NAMESVC)`) and fails fast if discovery breaks, so daemons never fall back to kernel-managed IDs or bootstrap handles. This keeps the privilege boundary clear: the kernel hands out the namesvc capability, and userland consumes every other service via the IPC API.

## Hardware bring-up

QEMU/KVM remains the reference environment, but we now track real hardware
logs (starting with a Dell OptiPlex 7020) to make sure ACPI, firmware memory
maps, LAPIC discovery, and serial paths behave the same on physical silicon.
See `docs/hardware.md` plus `boot-optiplex7020.log`/`docs/hardware-Dell7020.txt`
for the current dataset and the compatibility checklist used to evaluate new
x86_64 machines.

Both the UEFI and legacy boot paths now rely on an external ext2 root filesystem
image instead of embedding 4 MiB of disk data inside the kernel. The build
creates `build/rootfs.ext2` from the contents of `disk/ext2root/`, then:

- copies it into the FAT ESP as `EFI/BOOT/ROOTFS.IMG` (consumed by the UEFI
  loader), and
- bundles it into the Multiboot2 ISO as `/boot/rootfs.img` (loaded via
  `module2 … rootfs` in `boot/grub/grub.cfg`).

Whenever you change files under `disk/ext2root/`, rebuild the image with
`make ARCH=<arch> DEBUG=1 build/rootfs.ext2` (or just rerun your usual build,
which depends on the image). Boot logs will then reflect the updated contents,
and the ext2 fs-server will see a consistent rootfs across QEMU and physical
hardware.
`scripts/stage-rootfs.sh` is run automatically by both the ESP builder and
`scripts/mkiso.sh` so the FAT layout (`build/EFI/BOOT/ROOTFS.IMG`) and the GRUB
module (`/boot/rootfs.img`) always reuse the exact bits from `build/rootfs.ext2`.
The helper drops `build/rootfs.manifest` / `build/rootfs.iso.manifest` with a
SHA-256 sum of the staged image, making it trivial to confirm both boot paths
loaded the same payload or to restage manually while debugging (`./scripts/stage-rootfs.sh build/rootfs.ext2 build/EFI/BOOT/ROOTFS.IMG`).
Kernel boot logs now include `* x86_64_acpi:` sections that dump MADT/FADT/DMAR
content (LAPIC/IOAPIC entries, SCI overrides, VT-d ranges), so drop every new
hardware capture into `boot-<platform>.log` and keep the matching inventory in
`docs/hardware-<platform>.txt` for future audits.

### Block devices & virtio-blk

Two block devices are present today:

- **Device 0** – the loader-staged ramdisk built from `disk/ext2root/`. This
  exists so early bring-up still works on machines without virtio.
- **Device 1** – a paravirtualized virtio-blk disk backed by the exact same
  `build/rootfs.ext2` image. The user-mode `servers/block/virtio-blk/` daemon
  now owns this path entirely: it queries the PCI service for virtio-blk,
  maps the BARs via the hardware syscalls, allocates DMA via the new
  `SYS_dma_alloc`, spins up legacy virtqueues, and registers itself with
  blockd through the block-backend portal. Both `make run-uefi` and
  `scripts/run-qemu.sh` still attach the disk with
  `-device virtio-blk-pci,drive=virtio-rootfs`.

`servers/block/blockd` tracks every backend device that registers with it and
exposes their geometry via the block-backend portal. The ext2 server walks those
IDs via `block_get_info()` (which now proxies to blockd) and automatically picks
the largest device with 512-byte sectors (so the virtio disk wins when present).
If you want the ramdisk instead, either detach the virtio device or comment out
the QEMU drive flag.

The virtio-blk driver has now been fully evicted to user space. The
`servers/block/virtio-blk/` daemon launches at boot, queries the PCI service
for the virtio block function, inspects each BAR, maps the MMIO window via the
hardware-access syscalls, allocates queue/data buffers with the DMA syscall
pair, and services every request that blockd forwards through the backend IPC
channel.

To complete the split we introduced a dedicated block-backend IPC service:
`blockd` now registers its bootstrap portal via `block_backend_register()` and
device daemons resolve `"blockd"` via namesvc to register their geometry and
service I/O requests directly from user space.
`servers/block/ramdiskd/` and `servers/block/virtio-blk/` are the first backend
clients: ramdiskd registers a 4 MiB loader-backed disk, mirrors the contents
after mapping the loader’s rootfs with `hw_mmio_map()`, and services every
request synchronously, while virtio-blk drives the PCI device using its DMA
window and virtqueue logic. The kernel’s `SYS_blocksvc_request` path now simply
forwards requests to blockd/backends, keeping the block registry purely in
userland.

`kernel/block/core.c` is now a stub (kept only for early boot bookkeeping) so
the kernel no longer registers or serves block devices itself; every disk now
appears only when a user-mode backend daemon registers it with blockd.

Client code queries blockd’s registry directly: `block_get_info()` talks to the
backend portal so enumerators like ext2 can discover user-mode devices (IDs >1)
without touching kernel tables.

## Built-in demo & ELF loader

Until storage and a real userland loader exist, `kmain` boots a baked-in ELF
image that lives under `kernel/user/demo_image_*.h`. The image is generated
from the tiny assembly stubs in `user/demo_*.S` via the freestanding linker
script `user/demo_link.ld`, then embedded directly into the kernel binary.

At runtime the shared loader (`kernel/elf/loader.c`) validates the ELF header,
walks every PT_LOAD segment inside the user window, allocates pages via the
vm_space API, and seeds a guarded stack with `argc/argv/environ` stubs. The
demo launcher (`kernel/user/demo.c`) binds that vm_space to a scheduler task
and calls `arch_enter_usermode()` so syscalls can exercise the new dispatcher.

## User-mode tmpfs server

The first non-trivial user service ships alongside the demo. A second embedded
ELF (see `kernel/user/fs_image_*.h`) is staged and launched by
`kernel/user/fs_server.c`, which grants the task a bootstrap portal so it can
register with the kernel via `fs_register_service()`. Once registered, the
tmpfs loop blocks on `portal_recv()` and responds to open/read/close requests by
looking up the hard-coded file table in `servers/fs/tmpfs/main.c`.

Because everything runs in user space, IPC happens entirely through the
mailbox/portal path: the kernel forwards requests, the tmpfs task replies with
`fs_send_response()`, and the scheduler treats both sides like any other thread.
When you run `make run-legacy` or `ARCH=arm64 make run-arm64`, the fs-server
prints `[tmpfs] service registered` and then goes idle until the demo process
issues its first `open(2)`.

## User-mode console daemon

The physical console is now mediated by a tiny TTY daemon (`servers/console/ttyd/main.c`)
instead of the kernel. The console service forwards every completed keyboard
line to the daemon over a private portal (`console_tty_register()`), and the
daemon pushes those bytes into the active PTY via the new `pty_feed()` syscall.
This keeps stdin canonical mode and session ownership in user space while
preserving the existing `/dev/console` semantics for legacy tasks that still
call `console_open()`.

PTY 0 = Console (kernel commands: help, status, ptys, ticks, cpustats, perf, diag)
PTY 1 = fs-server
PTY 2 = ttyd
PTY 3 = ptyctl supervisor
PTY 4 = user-demo (echo demo)

## Repository layout: user apps vs. servers

Lenix keeps a strict split between “regular” user programs and long-running
microkernel servers:

- `user/` contains the freestanding runtime (`user/runtime/`) plus sample
  binaries such as the demo echo task (`user/demo/…`). These are the programs
  an administrator would treat like normal POSIX apps.
- `servers/` hosts every privileged user-mode service, organized by category:
  `servers/block/…` for storage drivers (`blockd`, future virtio devices),
  `servers/fs/…` for filesystems (tmpfs, ext2, vfs), `servers/console/…`
  for terminal daemons (`ttyd`, `ptyctl`), and `servers/drivers/…` for
  hardware-facing daemons such as the PCI enumerator. Each server is built into the
  kernel image as an embedded ELF (`kernel/user/*_image_*.h`) and launched via
  the corresponding `kernel/user/*_server.c` helper so the kernel never mixes
  server code with in-tree applications.

This layout keeps the microkernel boundary clear: production services evolve
under `servers/`, while `user/` remains a clean staging ground for ordinary
programs and the shared runtime.

After the demo prints “Type a line and it will be echoed back.” it enters a
userspace read loop. Since the QEMU run targets pipe the guest serial console to
your terminal, you can type characters and hit Enter to watch the user-mode
echo path exercise `read()`/`write()` via the syscall table. If you do nothing,
the VM simply waits for input—there is no kernel hang in that state.

## Runtime diagnostics

The interrupt-driven console accepts a couple of built-in commands so you can
inspect the live system without a debugger:

- `ticks` dumps the global jiffies/heartbeat counters exposed by the timer
  layer.
- `cpustats` prints per-CPU scheduler telemetry (ticks, switches, run-queue
  depth, current task).
- `perf` emits the Stage 6a performance counters (scheduler ticks/context
  switches, portal traffic, IPC byte totals, filesystem proxy activity, and the
  peak mailbox depth) so regressions are easy to spot while iterating on Stage 7.
- `blocktrace` dumps the last 32 block portal requests (device, op, LBA, blocks,
  bytes transferred, and jiffy duration) so you can diff UEFI vs Multiboot runs
  or capture the sequence that triggered a storage bug. Run `blocktrace clear`
  before reproducing an issue to capture a clean log for comparison.
- A kernel-owned console service logs when it starts and responds to basic
  commands (`help`, `status`) even while user-mode subsystems spin up, so
  you're never left staring at “Waiting for subsystems…”.
- `/dev/console` is now a portal-backed device. Call `console_open()` from
  `<lenix/console.h>` to grab a portal handle that can be `portal_recv()`'d for
  processed lines and `portal_send()`'d to print text without relying on `printk`.
- `diag` toggles runtime diagnostics: `diag status` prints the current state,
  `diag audit on|off`, `diag policy on|off`, and `diag telemetry on|off` control
  the verbose audit/policy logs and the periodic `[task:…] beat/jiffies` output.
- `pty_alloc()`/`pty_free()`/`pty_activate()`/`pty_query()` (see `<lenix/pty.h>`)
  let shells grab dedicated pseudo-terminals via the new kernel PTY mux. Every
  PTY exposes a portal for I/O, while `pty_activate()` switches the physical
  console to the desired PTY without rebooting.
- `pty_get_config()`/`pty_set_config()` adjust a PTY’s lightweight termios
  struct (canonical vs raw input, echo state, placeholder `rows/cols` window
  size), and `pty_send_signal()` injects stub `[signal N]` notifications to
  keep future SIGINT/job-control plumbing in sync.
- `stat()`/`fstat()`/`lseek()`/`dup()`/`fcntl(F_DUPFD|F_GETFL)` now route entirely through the tmpfs server: the kernel
  packages requests (`IPC_FS_REQ_STAT/FSTAT/SEEK/DUP/FCNTL`), tmpfs updates handle
  offsets and metadata, and the user runtime exposes the results via
  `<lenix/stat.h>`/`lseek(2)`/`dup(2)` so POSIX file queries never touch kernel filesystems.
- User-mode tasks write to stdout/stderr through per-task console portals
  instead of the kernel calling `serial_write()` on their behalf, which moves
  the console stack one step closer to a user-space TTY service.
- `Ctrl+N` talks to the new `ptyctl` supervisor task, which tracks every PTY via
  the `pty_enum()` syscall and cycles the foreground session (or accepts typed
  commands like `list`, `next`, `activate <id>` from its own PTY when you switch
  to it manually).

All commands run at interrupt level, so they are safe to invoke while the demo
waits for console input.