Sharpnr · Self-hosted data infrastructure
Infrastructure,
not business logic.
Silo Engine is Sharpnr's self-hosted platform for object storage, PostgreSQL provisioning, and data lake infrastructure. Organizations create the resources they need without ever touching the servers, RAID layout, or storage paths underneath them.
Self-hosted physical infrastructure. systemd and Nginx. Kubernetes deliberately out of scope for the first version.
The boundary
Sharpnr Engine owns business logic.
Silo owns infrastructure and data.
Silo is not the Hospitality, Accounting, or HR database. It is the layer those applications provision resources from — and the line between the two is the single most important decision in the architecture.
Silo Engine owns
- File storage & metadata
- Storage paths & backends
- Storage quotas
- Checksums & compression
- PostgreSQL provisioning
- Database credentials
- Database explorer APIs
- Data lakes & warehouses
- Data ingestion
- Backups & snapshots
- Audit logging
- Resource access control
Sharpnr Engine owns
- Hospitality
- Accounting
- HR
- CRM
- Other business modules
Business applications consume Silo resources. They do not reimplement them.
Platform
Three resources.
One foundation.
Storage, database, and data services are built on the same abstraction rather than as unrelated systems bolted together.
Files, objects, and CDN delivery on storage Silo owns.
Object storage
A URL that outlives
the filesystem.
Public addresses are deliberately independent of where the bytes physically live. Move to a different RAID array, a different backend, or a different datacentre — the URL does not change.
Public object
https://cdn.sharpnr.com/org_123/employee_456/profile/avatar.png
Quotas are enforced in Silo, not in the caller
StorageBackend
├── LocalRaidStorage // initial target
├── S3Storage
├── CephStorage
└── …
Checksums
Computed on the upload stream for integrity, dedupe, and audit.
Compression
Zstd first, behind a trait. Already-compressed formats are skipped.
PostgreSQL as a service
A database, not
a superuser.
Organizations receive scoped roles and a connection string. The physical cluster stays an implementation detail — and nobody gets superuser.
├── Create database
├── Create PostgreSQL roles
├── Apply permissions
├── Generate credentials
└── Return connection information
Returned
postgres://org_123_app:••••••••@silo-engine-db.sharpnr.com:5432/org_123_db
?sslmode=require
Sharpnr employees are application identities. Silo does not mint a PostgreSQL login for every employee that happens to exist.
No arbitrary SQL in v1.
The explorer exposes controlled operations over schemas, tables, views, functions, indexes, and constraints. A full SQL workspace comes later, once the controlled explorer is mature.
Browser → HTTPS → Silo Engine → PostgreSQL
Browser ⇸ PostgreSQL — never direct
Lakes & warehouses
Beyond a
drive clone.
Raw data lands in a lake, gets transformed, and becomes an analytical warehouse — all on the storage foundation Silo already manages.
Sources
PostgreSQL · files · events · jobs
Ingest
Scheduled & on-demand
Data Lake
raw / processed / curated
Transform
ETL / ELT to Parquet
Warehouse
Query & analytics
raw/
orders/ · customers/ · events/ · payments/
processed/
orders/ · customers/
curated/
daily_sales/ · customer_metrics/
lake://acme/orders/year=2026/month=08/day=10/
Partitioning, retention, and lifecycle are Silo's job — not yours.
Parquet is columnar and pairs well with analytical query engines, so it leads for analytical workloads.
Reuse, don't reinvent.
Silo leans on mature open-source engines rather than building an analytical database from scratch. DuckDB is the early candidate for querying Parquet directly; a distributed engine only arrives if the workload genuinely demands one.
Engineering
How it is
actually built.
A single Rust crate, an explicit dependency graph, and an OpenAPI contract at the edge.
Rust · Axum · Tokio
A single asynchronous service. Async tasks over the Tokio runtime — never an OS thread per request.
Storage behind a trait
LocalRaidStorage today; S3, Ceph, or anything else later. The public API does not change when the disks do.
Streaming everywhere
Reader → compressor → writer. Multi-gigabyte objects are never loaded whole into memory.
Explicit dependency injection
Config, pools, and backends are constructed once at startup and passed in. No DI framework, no service building its own dependencies.
One EngineConfig
Environment read once inside the config module, validated at startup, injected everywhere else.
OpenAPI as the contract
Swagger UI and ReDoc ship with the service. The spec is the interface between Silo and every consumer.
Append-oriented audit log
Durable infrastructure events with organization, actor, resource, and timestamp. Sensitive payloads stay out.
UUID v7 identifiers
Time-ordered by default, so identifiers index well without a separate sort key.
Design principles
- Infrastructure is separated from business logic.
- Storage sits behind a backend interface.
- Configuration is loaded once and injected.
- Public URLs never depend on physical filesystem layout.
- Large files are processed as streams.
- Distributed infrastructure arrives only when it is needed.
- Existing open source is reused before anything is reinvented.
The OpenAPI document is the contract between Silo and every consuming Sharpnr service.
Roadmap
Incremental,
on purpose.
Infrastructure is added only when the storage and operational model underneath it can carry the weight. Not every phase is committed to the first release.
Comparison
Where Silo fits —
and where it doesn't.
Silo is not trying to out-scale a hyperscaler. It is trying to make the infrastructure an organization already owns behave like a product. Phases mark what is built versus committed.
| Capability | Silo EngineSelf-hosted, one Rust service | Hyperscalere.g. S3 + managed Postgres | Backend platforme.g. Supabase-style | Self-assemblede.g. MinIO + Postgres + glue |
|---|---|---|---|---|
| Where the bytes live | Your hardware | Vendor region | Vendor or self-host | Your hardware |
| Object storage + CDN delivery | Phase 2–3 | Mature | Included | Assemble yourself |
| PostgreSQL provisioned via API | Phase 4 | Mature | Included | Manual |
| Controlled explorer, no raw SQL | Phase 4, by design | Not the model | Raw SQL editor | Bring your own client |
| Org + per-employee storage quotas | Phase 2, first-class | Bucket policy, not per-person | Not modelled | Build it |
| Data lake + warehouse on same storage | Phase 5–6 | Separate products | Out of scope | Build it |
| One audit trail across storage + DB | Phase 3 | Split per service | Partial | Build it |
| Egress billing | None — your network | Metered | Metered when hosted | None |
| Kubernetes required | No — systemd + Nginx | N/A, vendor-run | No | Often, at scale |
| Who carries the operational load | You run one binary | Vendor | Vendor or you | You run every part |
Phase labels refer to the Silo roadmap. Other columns describe categories of approach, not specific vendor SKUs.
Choose Silo when
- You already own the hardware and want the storage bill to stop being a line item.
- Quotas need to exist per organization and per employee, not per bucket.
- One audit trail should cover file access and database changes together.
- Business applications should provision infrastructure, not reimplement it.
Choose a hyperscaler when
- You need multi-region replication and elastic burst capacity today.
- There is no hardware and no team to operate it.
- Compliance requires certifications a self-hosted deployment cannot inherit.
Be honest: not yet Silo when
- You need a production data lake or warehouse this quarter — those are Phase 5 and 6.
- You want an arbitrary SQL workspace on day one; v1 is deliberately a controlled explorer.
- You only need one piece, and a single component you already run would do.
Questions
Frequently
asked.
Answers are drawn from the Silo Engine architecture document. If something here is out of date, the document is the source of truth.
What is Silo Engine?
Silo Engine is Sharpnr's self-hosted infrastructure platform. It provides file and object storage with CDN-backed delivery, PostgreSQL provisioning, a controlled database explorer, and — on the roadmap — data lakes, data warehouses, and analytics infrastructure. It is the layer Sharpnr Engine and other Sharpnr applications provision resources from.
Is Silo Engine available to use today?
Not yet. Silo Engine is in Phase 1, Foundation — the initial architecture and implementation phase. Configuration, the Axum server, application state and dependency injection, error handling, and tracing are the current focus. Storage lands in Phase 2 and PostgreSQL provisioning in Phase 4. Contact Sharpnr for early access.
How is Silo Engine different from Sharpnr Engine?
Sharpnr Engine owns business logic — Hospitality, Accounting, HR, CRM, and other business modules. Silo Engine owns infrastructure and data resources: storage, storage backends, quotas, checksums, compression, PostgreSQL provisioning, credentials, data lakes, backups, and audit logging. Silo is deliberately not the database for any business domain.
Which databases can Silo provision?
PostgreSQL. Silo creates the database, creates organization-scoped PostgreSQL roles, applies permissions, generates credentials, and returns connection information. The physical PostgreSQL server or cluster is an implementation detail.
Do organizations get PostgreSQL superuser access?
No. An organization never receives superuser credentials. Silo creates organization-scoped roles such as org_123_app, org_123_migrator, org_123_readonly, and org_123_admin. Sharpnr employees are application-level identities, so Silo does not create a PostgreSQL login for every employee.
Can I run arbitrary SQL in the Silo database explorer?
Not in v1. The explorer deliberately exposes controlled operations over schemas, tables, views, functions, indexes, constraints, and rows. An SQL workspace is planned for Phase 7, after the controlled explorer is mature. The browser never connects directly to PostgreSQL — it talks to Silo Engine over HTTPS, and Silo talks to PostgreSQL.
Does Silo Engine require Kubernetes or containers?
No. The initial deployment target is self-hosted physical infrastructure running as a normal service under systemd, with Nginx in front. Container orchestration is intentionally out of scope for the first version, though the architecture avoids assumptions that would block future horizontal scaling.
Where does Silo store data?
On RAID-backed storage on Sharpnr infrastructure — roughly 1 PB across multiple machines on a local network switch, with about 160 GB of combined RAM and 10 combined CPUs. Storage sits behind a StorageBackend abstraction, so S3, Ceph, or other implementations can be added without changing the public Silo API.
Are public file URLs tied to the filesystem layout?
No, deliberately. A public object is addressed as https://cdn.sharpnr.com/{org_id}/{employee_id}/{folder}/{filename}, which is independent of the physical storage path. The request travels from the public URL through CDN/Nginx to Silo Engine, then through the storage abstraction to physical storage.
What data formats does the Silo data lake support?
Parquet is the primary format under consideration because it is columnar and works well with analytical query engines. JSON, JSONL, CSV, Avro, and ORC are also planned. For early analytical and query workloads, DuckDB is a candidate for querying Parquet directly.
How does Silo handle very large files?
By streaming. Compression is designed as reader to compressor to writer so multi-gigabyte objects are never loaded whole into memory, with Zstandard as the initial algorithm behind a trait-based abstraction. Checksums are computed on the upload stream for integrity verification, duplicate detection, and auditing. Long-running work moves to background workers rather than blocking the request.
What is Silo Engine built with?
Rust, Axum, and Tokio, with PostgreSQL for metadata and OpenAPI as the published contract. Documentation is served at /swagger-ui, /redoc, and /v1/openapi.json. Configuration is centralized in a single EngineConfig loaded once at startup, and dependencies are injected explicitly rather than through a DI framework.
Infrastructure your organization creates, manages, and consumes.
Without ever managing the physical infrastructure underneath it. Silo Engine is early — talk to us about what you need from the storage and database layer.
Self-hosted · Rust · PostgreSQL · No Kubernetes required