Lightweight Single-System-Image Linux Cluster: A Project Proposal

Executive Summary

This proposal describes the design and implementation of a lightweight single-system-image (SSI) Linux cluster that allows multiple networked commodity Linux systems to appear and function as a single large multi-user timesharing system to end users. The primary goal is to provide transparent process execution, inter-process communication, and file system access across cluster nodes while maintaining the familiar Unix/Linux abstraction of a unified process space, global file namespace, and global user session view.

Unlike heavyweight SSI systems (e.g., ScaleMP's vSMP) that require high-performance interconnects and implement global shared memory, this system is designed to be lightweight, software-centric, and deployable on standard Ethernet-connected commodity hardware such as x86 servers or, as a proof-of-concept, a bank of Raspberry Pi 5 systems. The system is explicitly targeted at interactive multi-user workloads (shells, compilers, office applications, services) rather than tightly coupled HPC or large shared-memory applications.

This proposal outlines:

• The architectural design of the system and how it achieves SSI semantics

• Specific Linux kernel subsystems and system facilities that would require modification

• Implementation strategies drawing on prior SSI research (Kerrighed, openMosix, OpenSSI, Zap)

• Required expertise and organizational structure

• A realistic timeline for proof-of-concept and robust implementation phases

• Specific considerations for deployment on a Raspberry Pi 5 cluster

Problem Statement and Motivation

The Problem

Organizations often operate pools of independent Linux servers in order to aggregate compute, storage, and memory capacity. However, this approach creates operational friction:

• Users and administrators must be aware of multiple independent systems and explicitly distribute work across them

• Tools and workflows designed for single systems (interactive shells, pipes, file system navigation) require explicit distribution or adaptation for cluster use

• Session management, login tracking, and user administration must span multiple nodes manually

• Simple operations like "pipe process output from node A to process B on node C" require explicit network programming rather than transparent Unix pipes

For an interactive multi-user environment serving developers, researchers, or system administrators, the cognitive and operational load of managing multiple independent systems is substantial.

The Opportunity

A single-system-image cluster would eliminate this friction by presenting a unified, familiar Linux interface to users while transparently distributing work across physical nodes according to available capacity. A user would log in once, access what appears to be a conventional multi-user Linux system, and interact with processes, files, and other users without regard to physical node boundaries.

Such a system would be particularly valuable for:

• Organizations running shared development and build infrastructure

• Research clusters supporting interactive analytical work

• Educational institutions teaching parallel and distributed systems

• Cost-conscious deployments where commodity hardware (such as Raspberry Pi systems) can be aggregated without expensive high-speed interconnects

Background and State of the Art

Single-system-image clustering has been explored extensively in academic and commercial systems. Key prior work includes:

Academic and Open-Source SSI Systems

Kerrighed (2003–2010): A comprehensive SSI extension to the Linux kernel providing global process IDs, remote fork/exec, cluster-wide pipes via a "KerNet" distributed stream layer, and process migration. Kerrighed targeted academic research on cluster computing but did not achieve production adoption due to its reliance on mainline kernel integration and perceived complexity.[web:17]

OpenSSI / OSCAR (1998–2010): An open-source SSI cluster system that unified process space, file systems, and IPC across nodes, presenting a single large system image. OpenSSI demonstrated that SSI semantics could be practical for modest multi-user clusters, though active development ceased as commodity cloud infrastructure became prevalent.[web:9][web:11]

openMosix / MOSIX (1995–2005 for MOSIX; 2001–2012 for openMosix): An Israeli research system and its Linux implementation that provided transparent process migration and cluster-wide load balancing via kernel extensions and a distributed file system overlay (oMFS). openMosix is often cited as the closest precedent to a "lightweight" SSI for Unix systems, as it did not require a dedicated interconnect.[web:19][web:22]

Zap (OSDI 2002): A lightweight process migration and virtualization system for unmodified Linux applications, implemented as a loadable kernel module rather than full kernel modifications. Zap demonstrated that SSI-like behavior (process placement, virtualization of PIDs and network endpoints) could be achieved with minimal kernel integration, a principle valuable for this proposal.[web:23][web:20][web:25]

Commercial Systems

ScaleMP vSMP (2004–present): A commercial software virtualization layer that aggregates multiple x86 servers into a single NUMA shared-memory system. vSMP relies on a distributed hypervisor and requires high-speed interconnects (InfiniBand or similar) to maintain cache coherence and shared memory semantics, making it significantly more expensive and less suitable for interactive multi-user workloads.[web:6][web:1][web:10]

Gaps in the Literature

While the above systems demonstrate the feasibility of SSI clustering, each has limitations relative to the proposed design:

• Kerrighed, OpenSSI, openMosix are no longer actively maintained and were designed for prior generations of Linux kernels and hardware; their architectural patterns remain valuable, but a contemporary reimplementation with modern kernel infrastructure is justified.

• vSMP is expensive and over-engineered for interactive workloads; its focus on global shared memory and high-speed fabrics is unnecessary for the targeted use case.

• Zap focuses on process migration and pod mobility rather than continuous, seamless SSI semantics (global ps, who, pipes across nodes).

• None of the academic systems have been explicitly evaluated for deployment on low-cost, low-power commodity clusters such as Raspberry Pi systems, which are now sufficiently capable to serve as an interesting proof-of-concept platform.

This proposal fills that gap by proposing a modern, focused SSI implementation that:

1. Targets interactive multi-user Linux workloads on commodity hardware (Ethernet-connected servers or Raspberry Pi boards)

2. Maintains compatibility with standard Linux tools and workflows

3. Prioritizes simplicity, modularity, and minimal kernel changes (leveraging lessons from Zap and modern kernel practices)

4. Explicitly optimizes for interactive, latency-sensitive operations rather than HPC throughput

5. Avoids the global shared-memory complexity that drove vSMP's cost and hardware requirements

Proposed System: High-Level Design

Architecture Overview

The system consists of a small cluster of networked nodes (e.g., 4–16 commodity Linux systems) that cooperate via a cluster control layer to present a unified single-system-image to users and applications.

Key design principles:

1. Transparent process placement: Users invoke fork() and exec() normally; the cluster scheduler transparently places the new process on a node with sufficient capacity, while the user-space shell and libraries see a unified process space.

2. Extended PID space: Process IDs are logically extended to include a node identifier, allowing up to millions of globally unique PIDs. User-space tools like ps are adapted to display these global PIDs as if they were local.

3. Distributed pipes and IPC: When a Unix pipe connects processes on different nodes, the kernel transparently implements the pipe using a reliable network stream (TCP, SCTP, or a custom datagram-based protocol) instead of a shared buffer.

4. Global file namespace: A cluster file system or carefully configured NFS presents a single, unified file tree; data locality is a runtime hint, not a requirement.

5. Unified session and login view: A cluster login service (SSH to a frontend or load-balanced VIP) authenticates users once and registers their session globally; commands like who and w aggregate session information from all nodes.

6. Cluster awareness in resource management: The scheduler respects load, CPU availability, and data locality when placing processes; users do not explicitly choose nodes.

System Components

Cluster Node Architecture

Each physical node runs a standard Linux kernel extended with a cluster-aware module (or patch set) that:

• Extends the PID space to include a node identifier

• Implements remote fork/exec to place processes on chosen nodes

• Virtualizes pipes, Unix-domain sockets, and FIFO operations to work transparently across nodes

• Interfaces with a cluster messaging layer for inter-node communication

Cluster Messaging Layer

A reliable, efficient inter-kernel transport provides:

• Node membership and health monitoring (membership service)

• Process lookup and management (distributed PID registry)

• Remote system call forwarding (for syscalls that cannot be transparently handled locally)

• Distributed pipe and stream management

• Session and user registry synchronization

This layer is implemented as a combination of:

• A kernel module running on each node (for low-latency, high-throughput messaging)

• A lightweight user-space helper daemon for membership, higher-level RPC, and session management

• A simple, reliable protocol over Ethernet (e.g., a custom reliable datagram protocol or TCP-based messaging)

Process Scheduler

A distributed scheduler component (one instance per node, with a designated leader for global decisions) uses:

• Real-time load reports from each node (CPU, memory, I/O wait)

• Process placement policies (prefer local, consider data locality, respect user affinity)

• A remote fork/exec API to place processes on chosen nodes

Initial implementation prioritizes local placement (to minimize remote syscalls); later versions can add process migration for load balancing.

Global File System

A clustered file system (NFS with careful tuning, or a more sophisticated cluster FS such as CephFS, GlusterFS, or a simplified custom solution) provides:

• A unified mount tree visible on all nodes

• Coherent caching and eventual consistency appropriate for interactive workloads

• Locality hints so the scheduler can prefer running processes on nodes with local storage access

Login and Session Management

A cluster-wide login service:

• Fronts the cluster with a single hostname (e.g., cluster.example.com)

• Forwards SSH sessions to chosen nodes (or binds a frontend node as the login host)

• Maintains a global session table recording which users are logged in to which nodes

• Provides data to user-space tools (who, w, etc.) that aggregate session information

Core Modifications to Linux Subsystems

1. Process Model and PIDs

Current behavior: Linux PIDs are 32-bit integers, with 16 bits historically available for per-user PID allocation.

Proposed modification:

• Define a "cluster PID" (CPID) as a composite identifier: typically 22 bits = 6 bits node ID (up to 64 nodes) + 16 bits local PID.

• In the kernel, maintain a mapping table between CPIDs and local (node_id, local_pid) tuples.

• User-space libraries (libc) and tools (ps, kill, etc.) interpret CPID values transparently; applications continue to use standard getpid() and kill() syscalls.

• The kernel's internal process structures continue to use local PIDs for efficiency; the CPID abstraction is primarily for inter-node references and user-facing tools.

Implementation strategy: Modify the process creation code path to assign CPIDs; provide a /proc/cpids file or a get_cpid() syscall for userspace to query the mapping.

2. Remote Fork and Process Placement

Current behavior: fork() and execve() create and execute processes on the calling node.

Proposed modification:

• Introduce a cluster-aware fork mechanism (e.g., cluster_fork_on_node(node_id) as a new syscall or a libc wrapper around clone() and RPC).

• When a user process invokes this, the kernel forwards the fork request to the specified remote node; the remote kernel creates the child process and returns its CPID.

• The shell, sshd, and other critical daemons are linked against a wrapper libc that redirects fork() calls to the cluster-aware variant, using a placement policy (prefer local, or ask the scheduler).

• From the user's perspective, fork() works normally; the actual placement is invisible.

Implementation strategy: Implement as a loadable kernel module (for modularity) or as an in-kernel extension, following Zap's approach of virtualizing process creation without requiring a full hypervisor.

3. Pipes, FIFOs, and Distributed Streams

Current behavior: pipe() creates a local in-kernel buffer; data flows between processes in kernel memory.

Proposed modification:

• Extend the VFS layer to detect when a pipe has endpoints on different nodes.

• When detected, back the pipe with a "distributed stream" abstraction that:

  • Maintains a local FIFO buffer for same-node data flow (unchanged, for performance).

  • When writing to a remote endpoint, forwards data via a reliable network stream (implemented in the cluster messaging layer).

  • From the application's perspective, read() and write() on the pipe work normally.

• Implement distributed streams in the cluster messaging layer using either:

  • TCP sockets (simple, well-understood, but involves the network stack's overhead)

  • A custom reliable datagram protocol over Ethernet (more efficient, but requires careful design)

  • SCTP (StreamControl Transmission Protocol), which offers reliable, message-oriented delivery

Implementation strategy: Implement the distributed stream abstraction in the cluster module; hook into the VFS pipe creation and operation code to detect remote endpoints and redirect I/O appropriately.

4. Unix-Domain Sockets and Distributed Endpoints

Current behavior: Unix-domain sockets are addressed by a file path; connection state is maintained by the kernel.

Proposed modification:

• Extend the Unix-domain socket VFS implementation to:

  • Detect when a socket is being accessed from a different node (e.g., connect() to /var/run/some.sock, but the socket is on node B).

  • Transparently forward the connection request to the remote node's kernel via the cluster messaging layer.

  • Maintain a distributed endpoint abstraction analogous to distributed streams for pipes.

Implementation strategy: Modify the af_unix socket family implementation in the kernel to route remote connections through the cluster messaging layer.

5. Procfs and Process Visibility

Current behavior: Each node's /proc filesystem shows only local processes.

Proposed modification:

• Augment /proc or provide a separate /cluster/proc that:

  • Displays all cluster processes, tagged with their CPID and node ID.

  • Aggregates per-node data (e.g., /cluster/proc/stat combines CPU statistics from all nodes).

• Update user-space tools to query /cluster/proc by default:

  • ps displays all processes.

  • kill can target CPIDs across the cluster.

  • Shells' job control can display remote jobs.

Implementation strategy: Implement as a virtual file system (VFS) that queries the cluster messaging layer on each /proc access; cache results locally to avoid excessive network traffic.

6. Login, Sessions, and TTYs

Current behavior: Login occurs on a specific node; session state is recorded in /var/run/utmp and /var/log/wtmp locally.

Proposed modification:

• Establish a "frontend" node or a virtual entry point (e.g., via load balancing) where SSH connections arrive.

• The login process (sshd and login) authenticates the user and optionally spawns a shell via the cluster fork mechanism on a chosen node (or the frontend itself).

• A cluster-wide session service maintains a distributed /var/run/utmp-like database recording:

  • Logged-in users, their login time, and their node affinity (if any).

  • TTY allocation across nodes.

• Tools like who, w, users, and last query this distributed database.

Implementation strategy: Implement the distributed session service as a user-space daemon on each node (or on a few designated nodes) backed by a simple consensus protocol (or a primary/backup replica for simplicity); hook sshd and login to register sessions with this service.

7. File System and Global Namespace

Current behavior: Each node has its own mount tree; NFS mounts are per-node.

Proposed modification:

• Configure all nodes to mount a shared home directory tree (e.g., /home via NFS or a cluster FS).

• If possible, expose a cluster-wide view of storage:

  • A /cluster/storage/<node_id> namespace showing local disks of each node, or

  • A unified /data tree backed by a distributed FS (e.g., CephFS, GlusterFS).

• Implement locality hints in the scheduler:

  • If a process is about to read a large file from /data/projectX, and projectX is primarily stored on node 3, prefer placing the process on node 3 (or a node with network-local access to node 3's storage).

  • Use file system metadata (inode attributes, extended attributes) to tag data with preferred nodes.

Implementation strategy: Use standard NFS for the initial proof-of-concept; upgrade to a clustered FS for the robust phase. The locality hinting is implemented in the scheduler's decision logic.

8. Signaling and Process Control

Current behavior: kill() targets a local PID; signals are delivered via the kernel.

Proposed modification:

• Extend kill() to accept a CPID; the kernel detects if the target is remote and forwards the signal via the cluster messaging layer.

• Signal delivery to remote processes is handled by the target node's kernel.

Implementation strategy: Modify the kill() syscall handler to check the node ID in the target CPID; if remote, forward via RPC; if local, proceed normally.

Cluster Messaging and Communication Protocol

The reliability and latency of inter-node communication directly impacts user experience. The messaging layer should:

• Reliability: Guarantee delivery of messages (no silent drops); detect node failures and partition changes.

• Latency: Sub-millisecond message roundtrip times for interactive operations (pipes, process lookups).

• Throughput: Support high-bandwidth data flows (e.g., large pipes transferring GBs).

• Scalability: Handle clusters of 4–16 nodes initially; design for growth to ~64 nodes if needed.

Proposed implementation:

1. Protocol: A custom reliable datagram protocol over Ethernet, similar in spirit to SCTP or Kerrighed's KerNet:

   • Each message includes a sequence number; out-of-order or duplicate messages are detected and reordered/deduplicated in the receiving kernel.

   • Acknowledgments are sent for each message; timeout-based retransmission handles packet loss.

   • A heartbeat mechanism detects node failures and network partitions.

2. Kernel module: Implement the protocol in a loadable kernel module to avoid large changes to the core network stack. The module exposes a simple API to other kernel subsystems:

   • cluster_send_message(node_id, message) - send a message to a remote node.

   • cluster_register_handler(message_type, handler) - register a handler for a message type.

   • Incoming messages are dispatched to registered handlers via a work queue.

3. User-space daemon: A small daemon on each node handles:

   • Cluster membership and node discovery (e.g., via a configuration file listing all nodes, or a simple broadcast-based discovery).

   • RPC requests that are too complex or high-latency for the kernel (e.g., file system operations, session management).

   • Logging and diagnostics.

4. Performance tuning:

   • Use interrupt-driven message delivery if possible (to minimize latency).

   • Batch small messages to reduce packet count.

   • Ensure that messages are sent over a dedicated network (or VLAN) separate from user data traffic to avoid congestion.

Specific System Calls and Kernel Subsystems Affected

Proof-of-Concept Phase (6–12 months)

Goals

Demonstrate that a lightweight, software-only SSI cluster is feasible and practical for interactive multi-user workloads.

Deliverables

1. Kernel module and extensions (estimated: 2000–3000 lines of C code)

   • Extended PID space with CPID mapping.

   • Remote fork/exec mechanism.

   • Distributed pipe abstraction.

   • Cluster messaging layer.

2. User-space tools (estimated: 500–1000 lines)

   • Modified ps to query cluster process list.

   • Cluster session service daemon.

   • Scheduler component for process placement.

3. Test environment

   • A cluster of 4–8 Linux nodes (commodity servers or Raspberry Pi 5 boards).

   • NFS-based shared home directory.

   • Integration tests demonstrating:

     • Transparent cross-node pipelines (cat file | grep pattern | wc with stages on different nodes).

     • Remote process creation and execution.

     • Global process listing and job control.

     • Multiple concurrent users logging in and running shells.

4. Documentation

   • Architecture overview and design document.

   • Installation and configuration guide.

   • Known limitations and future work.

Scope Limitations (PoC Phase)

• No process migration: Processes are placed at creation time; no live migration.

• Limited IPC coverage: Pipes and basic Unix-domain sockets; no SysV IPC (shared memory, message queues) across nodes.

• Coarse failure handling: Node crashes or network partitions may result in orphaned processes; no automatic recovery.

• Single login node: Users log in to a designated frontend node; not yet load-balanced or replicated.

• File system: Standard NFS; no custom cluster FS or locality optimization.

Success Criteria

• A user can log in to the cluster via SSH to a single hostname.

• ps shows all processes across all nodes.

• who shows all logged-in users.

• A command chain such as find /data -name '*.log' | xargs grep ERROR | sort | uniq -c transparently distributes the pipeline stages across nodes and produces correct output.

• At least two users can run concurrent interactive shells and job control operations.

• Network latency for pipes and remote processes is acceptable (< 10 ms per roundtrip) on Gigabit Ethernet.

Robust Implementation Phase (2–4 years)

Goals

Harden the system for practical deployment in real environments, extend IPC and file system support, and add optional process migration.

Deliverables

1. Enhanced kernel subsystems

   • Robust distributed pipes and Unix-domain sockets.

   • Signal delivery to remote processes.

   • Process migration (optional, for load balancing).

   • Better TTY handling and terminal control.

   • Integration with systemd and modern Linux service management.

2. Advanced file system support

   • Integration with a clustered file system (CephFS, GlusterFS, or a custom design).

   • Locality hints and the scheduler respecting data affinity.

   • Transparent file migration or replication policies.

3. Cluster management and operations

   • Node membership and failure detection with automatic recovery.

   • Cluster configuration management (orchestration tools like Ansible or Terraform integration).

   • Monitoring and diagnostics (integration with Prometheus, Grafana, etc.).

   • Operational runbooks and documentation.

4. Security and isolation

   • Ensure that remote syscalls respect the calling user's UID/GID.

   • Implement message authentication (to prevent spoofing between cluster nodes).

   • Namespace isolation (users' processes cannot see or interfere with other users' remote processes).

5. Performance optimization

   • Profile and optimize the critical path (process creation, pipe I/O).

   • Caching strategies for distributed /proc lookups.

   • Load balancing and placement heuristics.

Success Criteria

• The system is deployable by a system administrator with standard Linux knowledge (no kernel hacking required).

• Automatic node failure detection and recovery.

• Process migration for load balancing (optional but demonstrated).

• Integration with standard Linux monitoring and orchestration tools.

• Sustained use in a real multi-user environment (e.g., a research lab or shared development infrastructure) for at least 12 months without major stability issues.

• Documentation is comprehensive and target audience is informed enough to extend and maintain the system.

Required Expertise

Successful implementation requires a multidisciplinary team:

Core Team (4–6 people)

1. Senior Linux Kernel Engineer (1–2 people)

   • Deep expertise in Linux process management, VFS, IPC, and scheduler.

   • Experience with kernel modules, subsystem hooks, and patching.

   • Familiarity with distributed systems concepts (eventual consistency, failure detection).

   • Rationale: The kernel modifications are the centerpiece; quality and correctness here determine success.

2. Distributed Systems Engineer (1 person)

   • Expertise in distributed consensus, membership protocols, fault tolerance.

   • Experience with reliable messaging, replication, and partition handling.

   • Familiarity with operational challenges (monitoring, debugging distributed systems).

   • Rationale: The cluster messaging layer, membership service, and failure handling require careful design.

3. Systems Engineer / Performance Specialist (1 person)

   • Profiling, tracing, and performance tuning of kernel and network subsystems.

   • Experience with latency measurement and optimization.

   • Understanding of hardware limitations and bottlenecks.

   • Rationale: The system will live or die by latency; this role ensures acceptable interactive performance.

4. DevOps / Operations Engineer (1 person)

   • Experience building, packaging, and deploying Linux systems.

   • Familiarity with CI/CD, automated testing, and containerization.

   • Scripting and infrastructure-as-code.

   • Rationale: Packaging, testing, and deployment must be robust from day one.

Supporting Roles

• Technical Writer (0.5 FTE): Documentation, design docs, operational runbooks.

• QA / Test Engineer (1 person, PoC phase; up to 2 in robust phase): Test planning, test automation, CI integration.

Skill Requirements

• All core team members should have:

  • At least 5+ years of Linux kernel or systems programming experience.

  • Comfort with C and low-level programming.

  • Experience reading and understanding kernel source code.

  • Ability to debug complex issues using tools like strace, perf, gdb, kernel printk debugging.

• Distributed systems expertise is critical; the messaging layer and membership protocol are the most complex and fault-prone components.

Implementation Strategy and Architecture

Modular Design

To minimize disruption to the Linux kernel and ease adoption, the implementation should follow a modular approach:

1. Kernel Module (the SSI layer)

   • Implements most of the CPID abstraction, remote fork/exec, distributed pipes, and integration points with the kernel.

   • Designed as a loadable module that can be inserted and removed without rebuilding the kernel.

   • Registers hooks with key VFS, process, and IPC subsystems to intercept operations.

2. Kernel Patches (minimal)

   • Only essential modifications to core subsystems (e.g., small hooks for the module to register).

   • Ideally, patches are small and backportable to multiple kernel versions.

   • Goal: keep the "core kernel" changes minimal so that future mainline kernel upgrades don't require extensive rework.

3. User-Space Daemons (libcluster, scheduler, session service)

   • Implement non-critical, high-latency components in user space.

   • Easier to debug, update, and extend.

4. Library and Tool Integration

   • A wrapper libc (or LD_PRELOAD library) that intercepts fork/exec calls and redirects them to the cluster layer.

   • Updates to standard tools (ps, who, etc.) to query the cluster state.

Build and Testing Strategy

• Continuous Integration:

  • Automated kernel builds and basic functional tests on every commit.

  • Tests include: spawning remote processes, creating pipes across nodes, checking process listing, etc.

  • A physical test cluster (or virtualized cluster using QEMU/KVM) to validate real networking.

• Regression Testing:

  • Ensure that existing local-only operations are unaffected by the cluster layer.

  • Performance regression tests to catch unexpected slowdowns.

• Integration Testing:

  • Multi-node scenarios with concurrent users, process chains, failure injection, etc.

Timeline and Resource Allocation

Proof-of-Concept Phase: 6–12 months (3–4 people full-time)

Estimated effort: ~2000–3000 person-hours for PoC.

Resource allocation:

• 1 senior kernel engineer (full-time throughout)

• 1 distributed systems engineer (50% starting month 2, full-time months 3–8)

• 1 systems engineer (starting month 4, 50% then full-time)

• 1 QA/ops engineer (starting month 5, 50%)

Robust Implementation Phase: 2–4 years (4–6 people, varying)

Estimated total effort: ~8000–12000 person-hours for robust implementation.

Comparison with Existing Systems

vs. Kerrighed / OpenSSI

vs. vSMP

vs. Zap

Example Proof-of-Concept: Raspberry Pi 5 Cluster

A Raspberry Pi 5 cluster is an excellent, practical testbed for this system. The specific characteristics of Raspberry Pi 5 hardware provide useful insights into design tradeoffs.

Hardware Profile

CPU & Memory:

• Quad-core Arm Cortex-A76 @ up to 2.4 GHz (roughly 2–3× the performance of Pi 4).

• 16 GB RAM per node (sufficient for the SSI layer, distributed services, and a modest number of user processes).

• Per-core performance is lower than contemporary x86 servers; designing for many small, I/O-bound processes is necessary.

Storage:

• NVMe boot via PCIe 2.0 x1 now practical and recommended over microSD; provides much better performance than SD.

• PCIe bottleneck (single lane) limits throughput; suitable for light I/O workloads, not high-bandwidth storage.

Networking:

• True Gigabit Ethernet (not over USB); measured at or near line-rate.

• Adequate for remote pipes and distributed file systems, but cross-node pipelines will see ~1 Gbps throughput and latency bounded by Ethernet physics (~0.1–1 ms per hop).

Power and Cooling:

• Low power draw (< 20 W sustained); active cooling recommended but not mandatory.

• Multiple boards are practical to deploy; reboots and hardware failures are more frequent than in data centers, so the system must be tolerant.

Design Implications

1. Process Placement and Locality

• The CPU per node is modest; avoid placing CPU-heavy workloads on remote nodes (latency and scheduler overhead will dominate).

• The scheduler should strongly prefer keeping related processes (e.g., a pipeline) on the same node when possible.

• If a process needs data, try to place it on a node with local or network-local storage access.

2. Distributed I/O and Pipes

• Gigabit Ethernet is bottleneck for large data flows; interactive shell pipelines (< 1 MB/s) work well.

• Cross-node pipelines incur ~1–5 ms latency per hop; acceptable for interactive use but not for tightly coupled, low-latency operations.

• Design the distributed pipe mechanism to prioritize latency over throughput; use direct kernel-to-kernel messaging rather than socket layers when possible.

3. Storage and File Systems

• A shared NFS server (one or two dedicated Pi nodes with large external NVMe storage) is practical.

• Alternatively, use a distributed FS (e.g., GlusterFS on commodity hardware) replicated across 3–4 nodes for durability.

• Expect per-node storage to be smaller; design the cluster to share a central /home and /data tree.

4. Failure Tolerance

• Raspberry Pi boards are prone to SD card corruption, power loss, and Ethernet glitches.

• The cluster membership and messaging layer must be robust to transient failures.

• Automatic node removal (if heartbeat is lost) and recovery (on restart) should be straightforward.

5. Performance Targets

• Process creation latency: < 50 ms for a remote fork/exec (vs. ~5 ms local).

• Pipe latency: < 10 ms per read/write roundtrip (vs. ~1 µs local).

• These targets are achievable on Gigabit Ethernet with careful kernel design; test early to validate.

6. Operational Simplicity

• Raspberry Pi OS is Debian-based with good mainline kernel support.

• Custom kernel patches can be maintained as a patch series applied to the standard Pi kernel tree.

• A single git repository + ansible playbook should suffice to build, configure, and deploy a cluster.

Reference Topology

A small proof-of-concept cluster might consist of:

• 4–8 Compute nodes: Raspberry Pi 5 (16 GB), each with NVMe boot drive, on a Gigabit switch.

• 1–2 Storage/frontend nodes: Pi 5 with large external NVMe or HDD storage, serving /home and /data via NFS/distributed FS.

• 1 Cluster manager node: May run on one of the compute nodes (or a separate small Pi 4); handles membership, logging, monitoring.

• Ethernet switch: Simple Gigabit, no special features needed.

Total cost: < $500–1000 in hardware (depending on storage capacity), making it an excellent sandbox for kernel development and cluster research.

Expected Limitations and Workarounds

Limitation: Single PCIe lane for NVMe limits storage throughput.
Workaround: Use external USB 3.0/3.1 storage for the shared data tier; network performance (Gigabit) is the true bottleneck, not storage.

Limitation: CPU per core is lower than x86; bursty workloads may exceed capacity.
Workaround: Provide burst capacity via a hybrid setup (Pi for baseline, x86 servers for peak); or teach users realistic expectations for this hardware tier.

Limitation: Longer latencies than data center clusters mean some latency-sensitive operations (e.g., distributed shared memory) won't work well.
Workaround: Accept that the system is optimized for interactive, not compute-parallel, workloads; document this clearly.

Advantage: Inexpensive and simple enough that many researchers and hobbyists can deploy and experiment with SSI concepts without significant capital investment.

Risk Analysis and Mitigation

Technical Risks

Organizational Risks

Mitigation Strategy

• Prototyping: Build minimal proof-of-concept early (first 3 months) to validate core assumptions.

• Community engagement: Present design at LKM or conference; solicit feedback; learn from prior SSI efforts.

• Modular architecture: Keep subsystems (messaging, scheduler, IPC) loosely coupled; allows parallel development and testing.

• Documentation: Record design decisions; maintain detailed code comments; this eases onboarding and reduces knowledge concentration.

Success Metrics and Evaluation

Proof-of-Concept Phase

1. Functional correctness:

   • ps on a frontend node shows all cluster processes (via CPID).

   • Cross-node pipelines produce correct output (data flows correctly through multiple nodes).

   • Multiple users log in; who and w show all sessions.

   • Process creation latency is < 100 ms for remote fork/exec.

2. Stability:

   • System runs without crashes for > 24 hours under typical multi-user load.

   • No data corruption or loss of messages.

3. Integration:

   • Existing tools (shells, compilers, grep, sort, etc.) work without modification on remote processes.

   • User experience is subjectively acceptable (latency < 1 second per operation).

Robust Implementation Phase

1. Operational readiness:

   • Documented deployment procedure; a sysadmin can stand up a new cluster in < 1 hour.

   • Automated node failure detection and recovery.

   • Integration with standard monitoring (Prometheus, Grafana) and logging (ELK, Splunk).

2. Scalability:

   • Cluster stability and performance with 16–32 nodes.

   • Sustained load test (e.g., 100 concurrent users, heavy compilation workload) for > 1 week.

3. Adoption:

   • At least 2–3 external organizations deploy the system in production.

   • Community contributions and active GitHub repository.

Conclusion

A lightweight, software-centric single-system-image Linux cluster is a feasible and valuable project that addresses real operational needs for shared compute infrastructure. By drawing on prior research (Kerrighed, openMosix, OpenSSI, Zap) and focusing on interactive multi-user workloads rather than HPC or shared memory, the system can be built with modest hardware requirements (commodity Ethernet-connected servers or Raspberry Pi boards) and a reasonable timeline (6–12 months for PoC, 2–4 years for robust implementation).

The key to success is a modular design that minimizes kernel changes, robust testing and failure handling, and clear scope management to avoid feature creep. The system will appeal to research institutions, small organizations, and hobbyists seeking to aggregate modest compute resources into a familiar, unified Linux environment.

A Raspberry Pi 5 cluster proves an excellent low-cost, practical testbed for validating the architecture and demonstrating feasibility to potential stakeholders and contributors.

References

[1] Healy, P. D. (2016). Single system image: A survey. Journal of Parallel and Distributed Computing, 28(1), 112–131. https://cora.ucc.ie/bitstreams/4b38c7e4-499d-438c-bf7d-3d00c2bfde29/download

[2] Single system image. Wikipedia. https://en.wikipedia.org/wiki/Single_system_image

[3] OpenSSI: OpenSSI Project. https://en.wikipedia.org/wiki/OpenSSI

[4] Kerrighed: Open-Source Cluster Middleware. Various publications and documentation (2003–2010). https://www.ssrg.ece.vt.edu/theses/MS_Ansary.pdf

[5] Bevilacqua, A., & Ferrorato, C. (2003). Kerrighed and Parallelism: Cluster Computing On Single System Image. Academic research documentation. https://www.scribd.com/document/352492934/01392625

[6] OpenMosix HOWTO. The Linux Documentation Project. https://tldp.org/HOWTO/html_single/openMosix-HOWTO/

[7] OpenMosix Project. SourceForge historical repository. https://openmosix.sourceforge.net/

[8] Osman, S., Subhraveti, D., Su, G., & Nieh, J. (2002). The Design and Implementation of Zap: A System for Migrating Computing Environments. Proceedings of the OSDI 2002, Boston, MA. https://www.cs.columbia.edu/~nieh/pubs/osdi2002_zap.pdf

[9] Zap: Process Migration. ACM Digital Library. https://dl.acm.org/doi/10.1145/844128.844162

[10] ScaleMP vSMP. Commercial product documentation. https://www.computer.org/csdl/proceedings-article/cluster/2010/4220a029/12OmNvnfkal

[11] ScaleMP vSMP for HPC Evaluation. FutureGrid Project Report (2011). https://www.infomall.org/sites/dsc/portal.futuregrid.org/projects/407.html

[12] Single System Image (SSI) Survey and Technical Overview. University of Melbourne Cloud Computing Group (2016). https://clouds.cis.unimelb.edu.au/papers/SSI-CCWhitePaper.pdf

[13] Introduction to Single System Image Cluster Technique. International Journal of Latest Trends in Engineering and Applied Mathematics, Vol. 3, Issue 4, 207–213 (2024). https://www.ijltemas.in/DigitalLibrary/Vol.3Issue4/207-213.pdf

[14] Ellis, A. (2023). First Impressions with the Raspberry Pi 5. Blog post. https://blog.alexellis.io/first-impressions-with-the-raspberry-pi-5/

[15] Ellis, A. (2023). Booting the Raspberry Pi 5 from NVMe. Blog post. https://blog.alexellis.io/booting-the-raspberry-pi-5-from-nvme/

[16] Raspberry Pi 5 Product Brief. Raspberry Pi Foundation. https://pip.raspberrypi.com/documents/RP-008348-DS-3-raspberry-pi-5-product-brief.pdf

[17] Tom's Hardware Raspberry Pi 5 16GB Review (2025). https://www.tomshardware.com/raspberry-pi/raspberry-pi-5-16gb-review

[18] Survey of Open Source Cluster Management Systems. Linux.com, September 2006. https://www.linux.com/news/survey-open-source-cluster-management-systems/