# I'm thinking about a multiprocessor system which works like this: there are multiple physical machines of the type that would run a regular linux system. They are network connected on a fast ethernet switch. A user logs in to 'the system' and sees a single multi-user linux system, not a single node in a set of multiple nodes. This is facilitated by several of the linux systems calls having been replaced by ones which behave as follows: process IDs (normally 16 bits) are extended to a larger size (say 22 bits) by making the PID a combination of the local PID and a machine node ID. Utilities such as 'ps' will see more entries when called as the extra entries come from the other nodes, but the user just interprets them as local PIDs. Linux command chaining via pipes can transparently work across machines because when piping data to another process, the system software will examine the PID of that process for its node component and if not on the same node as the source of the data, will implement the pipe functionality behind the scenes using a network mechanism instead, probably a socket. When a user starts a process, it will be started on their local node if there is capacity, or it may be started on a remote node, but transparently to the user.
There will be many linux system calls that can be transparently executed on other nodes, and some that cannot be done so efficiently (such as shared memory mapped files). The intention of this system is to supply a larger than normal multi-user service using smaller machines than typically needed. It is not for competing in the supercomputer cluster space.
If possible, users should be able to log in by ssh'ing to a single host name. A user would type 'who' and see all the logged in users regardless of which physical node they were connected to. Some type of networking mechanism would be needed to handle services running on the combined system.
Filing systems need to be handled transparently too, built on whatever is the best current technology available to implement this. When possible processes which access data directly should run on nodes that are attached to the drives with the data but locality is a runtime hint not a requirement.
I would like you to research the environment that would be needed and write a paper proposing writing such a system as a project, enumerating some of the linux system facilities that would need to be modified, with rough details of how they would be changed and which existing mechanisms the changes would be built on, what expertise would be needed, and how long the project might be expected to take to a) get a proof of concept and b) a robust implementation.
Note that there was once a system called vSMP by ScaleMP which presented a similar appearance of a single large system built from multiple nodes, but that was a much more heavyweight implementation that relied on high speed networking fabrics and which implemented virtual shared memory between nodes, which was expensive to implement. This proposal is a much more lightweight system that is primarily software and should not need significantly more expensive hardware than a pool of separate linux systems.
If there are other systems that implement parts of this proposal or are in some way similar to it, reference them and describe the significant parts.
This project is essentially a lightweight single-system-image (SSI) Linux cluster with a strong emphasis on transparent multi-user timesharing (UIDs, PIDs, TTYs, pipes, files) and only weak assumptions about shared memory and ultra-low-latency fabrics. Systems like MOSIX/openMosix, Kerrighed, OpenSSI and vSMP demonstrate that most of what you want is achievable with commodity hardware, but your proposal is intentionally narrower (no full global shared memory, focus on “looks like one box” for interactive/multi-user workloads).[^1][^2][^3][^4][^5]
## Related systems and prior art
There is a long history of SSI work on Unix-like systems that maps very directly to your wish-list.
- **openMosix / MOSIX**: Linux kernel patches that transparently migrate processes for load balancing; a process has a “home node” that handles its system calls, while the user-space image can execute on another node; provides an optimized distributed file system (oMFS) to give the illusion of a single namespace; designed so a cluster “looks like a big SMP system” to users.[^5][^6][^7]
- **Kerrighed**: A full SSI extension to Linux with global process IDs (“KPID”), remote process creation, remote fork/exec equivalents, global file descriptors, and a distributed “KerNet” layer that implements cluster-wide pipes and sockets; includes process migration and global memory containers.[^4][^8]
- **OpenSSI**: Another open-source SSI cluster that unifies process space, IPC, and file systems across nodes to present a single large system to users.[^3]
- **Zap**: Process-migration for unmodified Linux applications using a thin virtualization layer and “pods” (groups of processes) with virtualized PIDs, network endpoints, etc., implemented as a loadable kernel module rather than a full SSI.[^9][^10][^11][^12]
- **ScaleMP vSMP**: Commercial software that aggregates multiple x86 servers into a single virtual SMP with large shared memory using a distributed hypervisor approach; significantly more heavyweight, aimed at big shared-memory systems.[^13][^2][^14]
The proposed system is closest in spirit to Kerrighed/OpenSSI/openMosix, but with a design target of “transparent multi-user node farm” rather than large shared-memory HPC or full-blown SSI, and keeping the hardware fabric limited to fast Ethernet.[^1][^3][^4][^5]
## High-level architecture and environment
A practical architecture reuses existing mechanisms in Linux and in prior SSI work, while keeping the kernel changes focused.
- **Cluster model and naming**
- A small, fixed set of nodes, all running compatible Linux kernels and a cluster extension layer.
- One logical “cluster hostname” used for SSH and services; individual nodes retain private hostnames/addresses.
- Global node IDs maintained by a cluster manager; used for PID composition and for routing.[^3][^4]
- **Single system image elements**
- Global view of users/logins: commands like `who` and `w` aggregate session information from all nodes.[^3]
- Global process space: PIDs extended with a node component; tools like `ps` see all cluster processes.[^4]
- Global IPC and streams: pipes, Unix-domain socket pairs, and possibly FIFOs can transparently span nodes using a distributed pipe abstraction.[^5][^4]
- Global file namespace: either via a cluster file system (e.g., something like PXFS/OpenSSI’s approach) or a highly consistent NFS/clustered FS, possibly with locality hints.[^15][^5]
- **Scheduling and placement**
- A cluster scheduler performs initial process placement using load metrics from each node and an API for “remote fork/exec”.[^4][^5]
- Optionally, process migration can be supported later (similar to openMosix and Kerrighed), but initial implementation can confine itself to first-placement only and avoid the complexity of migration.[^16][^5][^4]
- **Networking**
- A reliable, reasonably low-latency Ethernet fabric (1–10 GbE) is sufficient, since you are not aiming for tightly-coupled shared-memory HPC.[^2][^14]
- A cluster messaging layer (kernel module plus small user-space helper) for: PID lookups, remote syscalls, global pipes, node membership, and cluster RPC. Kerrighed’s KerNet and openMosix’s internal messaging provide good reference designs.[^6][^16][^4]
## Kernel / system facilities to modify or extend
Several Linux subsystems would need explicit cluster-aware extensions. Existing SSI systems show viable patterns for each.
### Process model and PIDs
- **Extended PID space**
- Define a “cluster PID” that includes a node ID and a local PID; Kerrighed uses a KPID constructed from the initial Linux PID and the node identifier.[^4]
- Maintain a mapping between local PIDs and global CPIDs in the kernel; user-space tools can be updated to display CPIDs while the kernel uses local PIDs internally.[^3][^4]
- **Remote fork/exec and process placement**
- Provide a kernel API (or a thin user-space stub) that implements “fork on node N then exec”, semantically equivalent to `fork()` followed by `exec()` on that node, similar to Kerrighed’s remote process creation.[^8][^4]
- The shell and `login`/`sshd` can then create processes via this API, letting the cluster scheduler decide whether “local” means this node or a remote, lightly loaded node.[^5][^4]
- **Procfs and `/proc` visibility**
- Modify procfs to merge per-node process lists into a single global view, or provide a cluster-aware `/cluster/proc` that tools like `ps` use by default. OpenSSI and Kerrighed expose global process views in this way.[^3][^4]
### Pipes, sockets, and IPC
Transparent cross-node `pipe()` and related IPC is central to your design; Kerrighed and openMosix already demonstrate workable schemes.
- **Distributed pipes and streams**
- Replace the local pipe implementation with a cluster-aware pipe object; when both endpoints are local, behave normally; when endpoints reside on different nodes, back the pipe with a connection over the cluster messaging layer (a kernel-level reliable stream, or a kernel-managed TCP/UDP socket).[^4]
- Kerrighed’s KerNet implements “dynamic streams” and KerNet sockets to provide cluster-wide pipes and stream files; these can be a direct design template.[^4]
- **Unix-domain sockets and FIFOs**
- Unix-domain sockets can be virtualized similarly, with a name lookup that resolves to a node and creates a remote endpoint; KerNet’s “global stream management” is a precedent.[^4]
- FIFOs in a shared file system can be backed by distributed pipe endpoints once opened, just as local FIFOs map to pipes.[^15][^4]
- **Shared memory and mmap**
- Because the goal is lightweight and not full cluster-wide shared memory, shared memory segments should remain local to a node; attempts to share across nodes can either be disallowed or redirected to a slower global mechanism (e.g., a file in the global FS with advisory hints), similar to how some SSI systems restrict certain SysV IPC semantics across nodes.[^17][^3]
### Filesystems and storage
The user expectation is a unified file namespace, ideally with some notion of locality.
- **Global namespace**
- Use an existing clustered or network file system that can provide POSIX-like semantics cluster-wide: NFS with care, or a more SSI-friendly design like PXFS (used in Solaris MC) or OpenSSI’s global FS; these intercept VFS/vnode operations and redirect them to the node that holds the data.[^15][^3]
- Alternatively, adopt an openMosix-like /mfs tree exposing each node’s file system under a global mount point, combined with a cluster-wide `/home` and application directories.[^5]
- **Placement hints**
- Introduce an optional API for processes to declare data locality preferences (e.g., “prefer node with storage for /data/projectX”), used by the scheduler when placing new processes; this is comparable to the “process placement” heuristics in Kerrighed that consider CPU and temperature probes and memory locality.[^4]
### Login, sessions, and multi-user tools
To meet the “log in to the system, not a node” requirement, the login and session subsystems need cluster awareness.
- **SSH and logins**
- The cluster’s DNS entry points to a frontend node or a load-balancing VIP; the frontend can:
- Either be the permanent login node that then uses remote fork/exec for user shells on chosen compute nodes (similar to home node semantics), or
- Forward SSH sessions to chosen nodes based on load, while still registering sessions in a central cluster session service.[^5][^3]
- **`who`, `w`, `users`, TTYs**
- Maintain a cluster-wide session table; each node reports logins to a central daemon or distributed registry.[^3]
- Update user-space implementations of `who` et al. to query this registry instead of only local `/var/run/utmp`. OpenSSI and similar systems expose SSI-aware versions of these commands.[^17][^3]
### Cluster services and networking endpoints
Services should appear bound to the “system” rather than to a specific node.
- **Service abstraction**
- A cluster service manager maintains a mapping from service names to one or more instances on nodes; the “system” IP listens and forwards connections to service instances using a simple load-balancing or failover strategy.[^3]
- Existing technology such as LVS, HAProxy, or IPVS can provide the external facade, while the internal SSI layer manages processes and PIDs.[^2][^1]
- **Node management and membership**
- A node manager runs on each node, managing local processes and reporting node health and load; a global scheduler uses this information for placement. MOSIX/openMosix use a similar distributed scheme.[^6][^17][^5]
- A small consensus mechanism (even a simple primary/backup approach for this project) handles leadership for global decisions, though heavyweight distributed consensus might not be necessary.
## Required expertise and implementation effort
Building this as a real system is substantial, but much of the conceptual work has been explored previously.
- **Expertise needed**
- Linux kernel development: process management, scheduler, VFS, pipe/IPC, network stack, and netfilter familiarity.[^10][^5][^4]
- Distributed systems: membership, failure detection, cluster messaging, and basic consensus or leader election.[^17][^15]
- Systems and performance engineering: profiling cross-node IPC, understanding latency and throughput tradeoffs on Ethernet, ensuring that remote pipes and remote file access remain acceptable for interactive loads.[^5][^4]
- Toolchain/user-space integration: modifying `ps`, `who`, `sshd`, shells, and possibly PAM / session management to integrate with the SSI layer.[^5][^3]
- **Team profile**
- 1–2 senior kernel/OS engineers with prior experience in kernel subsystems or SSI-like projects.[^5][^4]
- 1 distributed systems engineer for the messaging and membership components.[^17]
- 1–2 systems engineers for packaging, automation, and operations (installing the cluster, building and testing kernels, CI).[^3]
- **Rough schedule**
These estimates assume a small, focused team (3–4 engineers) working full-time and piggy-backing on existing research designs without trying to generalize to “production-ready for everyone” in the first phase.
- **Phase A – Proof of concept (PoC): 6–12 months**
- Goals:
- Global PID abstraction and basic remote fork/exec.[^4]
- Minimal cluster messaging layer (kernel module plus user-space helper).[^5][^4]
- Cross-node pipes for simple command pipelines (`cat | grep | wc` with stages on different nodes). [^4]
- Global `ps`/`who` that aggregate across nodes.[^3][^5]
- A shared home directory via NFS or similar, with processes placed on remote nodes based on a rudimentary scheduler.[^15][^5]
- Expected outcome: cluster behaves like a single multi-user Linux box for simple interactive shell pipelines; no process migration yet; limited IPC coverage; coarse failure handling.
- **Phase B – Robust implementation: 2–4 years**
- Objectives:
- Hardened cluster messaging and membership with node failure handling and recovery.[^15][^17]
- More complete IPC and FS semantics (Unix-domain sockets, FIFOs, signal delivery to CPIDs, robust global /proc, better cluster FS integration).[^3][^4]
- Optional transparent process migration for better load balancing, drawing from openMosix and Kerrighed techniques.[^16][^4][^5]
- Integration with service management (systemd units aware of node-level and cluster-level states).
- Tooling, documentation, and operational practices so that it can be deployed by sysadmins with minimal kernel-fu.
- Expected outcome: a practical, robust SSI-like environment suitable for modest multi-user installations that prefer a “single big machine” abstraction over explicit clusters.
## Positioning and differences from existing SSI / vSMP systems
Relative to the prior art, this proposal is both more modest and more targeted.
- Compared to **vSMP**:
- vSMP aggregates nodes into a single large shared-memory SMP system by inserting a distributed hypervisor under the OS; it targets very large memory footprints and core counts with strong global memory coherence requirements, and expects high-performance fabrics.[^13][^2]
- The proposed system does not create a true NUMA machine; it keeps memory local to nodes, avoids global cache-coherent shared memory, and therefore does not require specialized interconnects, making it cheaper and simpler to deploy.[^1][^2]
- Compared to **Kerrighed / OpenSSI / openMosix**:
- These projects aim for a full SSI cluster, often including global memory abstractions and sophisticated process migration, and have broader IPC and FS semantics.[^4][^5][^3]
- This proposal can deliberately constrain scope:
- Focus on interactive multi-user workloads (shells, compilers, daemons) rather than HPC or tightly coupled parallel applications.[^5]
- Treat process migration as optional; initial versions only do remote placement.[^4][^5]
- Avoid cluster-wide shared memory beyond what a cluster FS and simple IPC can provide.[^17][^3]
- Compared to **Zap**:
- Zap focuses on migrating pods of processes across independent Linux systems, mainly for administration, load management, and availability; it virtualizes PIDs and resources using a loadable kernel module without a full SSI.[^11][^9][^10]
- The proposed system cares more about continuous SSI semantics (global `ps`, `who`, pipes across nodes) than about migrating whole computing environments, though some of Zap’s virtualization concepts could be reused for process migration and for keeping modules out of the core kernel.[^10][^4]
From a project-proposal perspective, you can frame this as a focused, modern reimagining of classic SSI clusters for contemporary commodity hardware and interactive workloads, leveraging prior designs (Kerrighed, openMosix, OpenSSI, Zap) while intentionally staying away from the complexity and hardware demands of vSMP-style global shared memory.[^2][^1][^5][^4]
[^18][^19][^20][^21][^22][^23][^24][^25][^26][^27][^28]
⁂
[^1]: https://en.wikipedia.org/wiki/Single_system_image
[^2]: https://www.businesswire.com/news/home/20200114005508/en/ScaleMP-Demonstrates-Best-in-Class-Large-System-Results
[^3]: https://en.wikipedia.org/wiki/OpenSSI
[^4]: https://www.scribd.com/document/352492934/01392625
[^5]: https://tldp.org/HOWTO/html_single/openMosix-HOWTO/
[^6]: https://krisbuytaert.be/howto/openMosix-HOWTO/c145.html
[^7]: http://fluid.ippt.gov.pl/mosix/TheopenMosix.pdf
[^8]: https://arxiv.org/pdf/1507.01067.pdf
[^9]: https://dl.acm.org/doi/10.1145/844128.844162
[^10]: https://www.cs.columbia.edu/~nieh/pubs/osdi2002_zap.pdf
[^11]: https://www.cs.cmu.edu/~sosman/talks/osdi2002/osdi2002_zap_talk.pdf
[^12]: https://www.usenix.org/conference/osdi-02/design-and-implementation-zap-system-migrating-computing-environments
[^13]: https://cora.ucc.ie/bitstreams/4b38c7e4-499d-438c-bf7d-3d00c2bfde29/download
[^14]: https://www.infomall.org/sites/dsc/portal.futuregrid.org/projects/407.html
[^15]: https://clouds.cis.unimelb.edu.au/papers/SSI-CCWhitePaper.pdf
[^16]: https://openmosix.sourceforge.net/linux-kongress_2003_openMosix.pdf
[^17]: https://www.ijltemas.in/DigitalLibrary/Vol.3Issue4/207-213.pdf
[^18]: https://dl.acm.org/doi/abs/10.1016/j.jpdc.2016.01.004
[^19]: https://www.computer.org/csdl/proceedings-article/cluster/2010/4220a029/12OmNvnfkal
[^20]: https://www.cs.cmu.edu/~sosman/publications/osdi2002/osdi2002_zap.pdf
[^21]: https://thesiliconreview.com/magazine/profile/expand-beyond-the-limits-with-scalemp
[^22]: https://www.eng.auburn.edu/ens/hpcc/vsmp-scale-mp.html
[^23]: https://www.linux.com/news/survey-open-source-cluster-management-systems/
[^24]: https://stovaris.pl/en/rozwiazania/pamieci-masowe-i-archiwizacja/scalemp-kompleksowe-rozwiazania-dla-hpc-skalowania-mocy-obliczeniowej-i-zwiekszania-pamieci/
[^25]: https://www.osti.gov/servlets/purl/891617
[^26]: https://www.ssrg.ece.vt.edu/theses/MS_Ansary.pdf
[^27]: https://core.ac.uk/download/pdf/147555755.pdf
[^28]: https://blog.csdn.net/jiayanhui2877/article/details/8230481
# Add a postscript that comments on any significant aspects of this design if implemented on a bank of 16Gb raspberry pi 5 systems.
A bank of 16 GB Raspberry Pi 5 boards is a plausible hardware target for this system, but it pushes different constraints than x86 servers: CPU, per-core performance, I/O bandwidth, and network latency will dominate design tradeoffs more than RAM capacity.[^1][^2][^3]
## Postscript: Raspberry Pi 5 cluster considerations
- **CPU and concurrency envelope**
- Raspberry Pi 5 uses a quad-core Arm Cortex‑A76 at up to 2.4 GHz with roughly 2–3× the CPU performance of a Pi 4, which is solid but still below contemporary x86 server cores in per-thread throughput.[^2][^4]
- For your SSI design, this means the global multi-user illusion is feasible, but the per-node process density for CPU-heavy workloads will be modest; the scheduler should bias toward many small, mostly I/O-bound processes instead of a few heavy compute jobs.[^3][^4]
- **Memory vs. I/O balance**
- With 16 GB RAM per node, memory will often be over-provisioned relative to CPU, so the cluster can host many lightly loaded sessions or daemons, but not many CPU-bound tasks per board.[^3]
- The large RAM capacity encourages aggressive page cache usage and avoids swap, but backing storage (microSD or NVMe via PCIe x1) will define actual FS behavior; Pi 5’s PCIe 2.0 x1 and SD SDR104 support can give decent SSD performance, yet still well below a typical server NVMe.[^5][^4][^6][^1]
- **Network and distributed pipes**
- Pi 5 has true Gigabit Ethernet, and reviewers measure it at or near line-rate, which is adequate for remote pipes and distributed FS, but cross-node pipelines for chatty workloads will still be noticeably slower than local pipes.[^7][^1]
- The SSI layer should:
- Prefer to keep tightly pipelined jobs (e.g., `grep | sort | uniq`) on the same node when possible.
- Use remote pipes for longer-running, less latency-sensitive flows, accepting that interactive feel across nodes is bounded by ~1 Gbps and Ethernet latency.[^4][^7]
- **Storage layout and locality**
- Booting from NVMe on Pi 5 via the exposed PCIe lane is now practical and gives much better I/O than SD, but still constrained by a single PCIe lane and 1 GbE when used as shared storage.[^6][^1][^4]
- A reasonable layout would use:
- Local NVMe per node for root and temp/scratch.
- A central or distributed NAS (possibly another Pi or two with NVMe) for user home directories and shared data, exposed via NFS or a cluster FS; your scheduler can then treat nodes with direct-attached data as preferred for data-heavy processes.[^8][^9][^4]
- **Energy, cooling, and density**
- Pi 5 boards have low power draw relative to x86 servers, making a “farm” of them attractive for experimentation, but active cooling is recommended at sustained high loads.[^1][^4]
- The low power and small form factor make it realistic to deploy many nodes; your SSI layer must be robust to frequent node reboots or SD/NVMe quirks that are more common in hobby-grade hardware.[^4][^6]
- **Software and kernel ecosystem**
- Raspberry Pi OS is Debian-based, with good mainline kernel support for Pi 5; this makes carrying custom kernel patches for SSI-like functionality realistic, though you’ll have to track the Pi-specific kernel tree or mainline depending on feature needs.[^5][^4]
- For a first proof-of-concept, a Pi 5 cluster is an excellent platform: inexpensive, homogeneous, and with enough CPU and RAM to exercise global PIDs, remote fork/exec, and distributed pipes under real multi-user loads, even if it won’t match server-class machines in raw throughput.[^3][^4]
[^10][^11][^12][^13][^14][^15]
⁂
[^1]: https://www.adafruit.com/product/6125
[^2]: https://www.raspberrypi.com/products/raspberry-pi-5/
[^3]: https://www.tomshardware.com/raspberry-pi/raspberry-pi-5-16gb-review
[^4]: https://blog.alexellis.io/first-impressions-with-the-raspberry-pi-5/
[^5]: https://pip.raspberrypi.com/documents/RP-008348-DS-3-raspberry-pi-5-product-brief.pdf
[^6]: https://blog.alexellis.io/booting-the-raspberry-pi-5-from-nvme/
[^7]: https://bret.dk/raspberry-pi-5-review/
[^8]: https://www.reddit.com/r/HomeServer/comments/1oo8zd4/raspberry_pi_5_m2_board_for_a_home_nas/
[^9]: https://thepihut.com/blogs/raspberry-pi-tutorials/how-to-flash-an-nvme-boot-drive-with-raspberry-pi-5
[^10]: https://www.pishop.us/product/raspberry-pi-5-16gb/
[^11]: https://www.newegg.com/p/3D0-0001-000K2
[^12]: https://www.reddit.com/r/admincraft/comments/1buuabf/has_anyone_hosted_a_server_on_a_raspberry_pi_5/
[^13]: https://www.youtube.com/watch?v=sFDn_TFbZpg
[^14]: https://www.youtube.com/watch?v=NmpcKv91LLs
[^15]: https://www.canakit.com/raspberry-pi-5-16gb.html