Stalwart is, from what I’ve read, an excellent JMAP server.
JMAP is, from what I’ve read, a great protocol for building an E-Mail (and now also others) client on top of.
Since I would like an innovative way to access my E-Mails, but do not want to self-host, I would find it interesting to use Stalwart as the server component of an E-Mail client: Data is somehow synced into Stalwart via the “ugly” protocols and I get a nice API to build an elegant client on top of.
My basic research shows that something like IMAP-IMAP sync seems to be a thing. Has anyone done something like this, perhaps even with Stalwart? (this of course grows in complexity for each new protocol to be proxied).
I believe having this kind of setup easily accessible could jumpstart a new generation of E-Mail clients on top of JMAP because it (relatively elegantly?) circumvents the chicken-egg problem by allowing all existing IMAP mailboxes to be accessed via JMAP.
I would just like to highlight the "excellent" bit here. It is a tour de force, a masterpiece. Beautifully factored software. They've done an excellent job incrementally building trust and delivering. Really cool project, and I'm so glad it exists.
I also did not realize it is… primarily developed by a single person? The author. Incredible:
This should be pretty straightforward to do with an IMAP <-> IMAP syncing tool, like mbsync [1]. You'd run it periodically in the background to sync the remote IMAP to Stalwart's local IMAP server, and Stalwart can then automatically serve that via JMAP, doing the translation internally.
I was originally thinking you'd need to go remote IMAP <-> maildir <-> Stalwart IMAP, which would be really complicated, but I think the IMAP <-> IMAP should work fine.
ImapGoose is on my radar to replace mbsync and imapnotify in my setup, but I think it's IMAP <-> maildir only, which makes sense for the intended use case of local mail
Reading weirdly many “but there are no clients” posts. Of course something has to come first. You can’t really develop a client without a server implementation and Stalwart is essentially the first server implementation of JMAP.
With Stalwart in place, there’s finally a reason to develop a client for JMAP.
I hope y’all are aware that Mozilla’s new mail service will use it, so that is likely going to give JMAP a big push!
I'm cautiously optimistic that Stalwart is a game-changer. I would normally stick with a simpler platform (like mox) if I wasn't interested in the groupware space. I've previously tried Nextcloud and SoGo and left disappointed, and have more or less been waiting for a project like this to come along.
Nextcloud was such a terrible experience for me (the file sharing/storage was good, but the groupware aspect was incredibly buggy). But knowing that Nextcloud is partnering with Stalwart to hopefully overhaul their stack, Opencloud is developing their JMAP integration, and Mozilla/Thunderbird is using it too (they already have a webmail in development here: https://github.com/thunderbird/stormbox)... we might finally see some exciting development in this space. And now is also a ripe time, as there seems to be a perfect storm of people wanting to get away from Big Tech platforms.
While JMAP seems to scratch every itch of a sucker for proper web API design, I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP. Is there really any new-ish binary protocol these days?
Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO. Then again, a lot of solid thinking went into these things, so there probably are a lot of good reasons that I’m not aware of.
Email was never a binary protocol. Notoriously so, it's why MIME types and MIME encodings get so complicated.
Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet. HTTP as the new telnet has a bunch of improvements when it comes to binary data, request/response-based data flows, and some other considerations. HTTP/3 is even inherently a binary protocol, it's lack of "telnet-compatibility" one of the concerns about switching the majority of the web to it.
vCard/vCal/iCard/iCal were also deeply "plaintext formats". JSON is an improvement because it is more structured, even more efficient, than those predecessors. JSON may not look efficient, but it compresses extremely well and can be quite efficient in gzip and Brotli streams.
I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet", even if it doesn't sound like "binary protocol efficiency" at first glance. Especially as HTTP/3 pushes HTTP more efficient and more "binary", and arguably "more fundamental/basic" with HTTP/3 even taking over more roles in the TCP/UDP layer of the internet stack. (Telnet would never try to replace TCP.) HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of. Sure, it would be neat to see more variety and experiments outside of the HTTP stack, too, but HTTP is too useful at this point not to build a bunch of things on top of it instead of as their own from-scratch protocol.
A lot of the textual nature of older IETF protocols, including the CR LF line endigns, can be probably traced to how easy it was to bang out a bad implementation full of subtle problems that could be debugged by sitting an undergrad student at a teletype instead of spending time on having some binary serializer (that telecom companies definitely had money for)
A lot of old RFCs explicitly mention running on top of TELNET.
Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and one of the earliest ways to access ARPAnet outside of machines directly on it was through calling into a TIP which set up bidirectional stream from your modem to a port on some host.
> Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and
The idea with packets is that you don't need to reserve N bit/s of each link along the route to whatever system you're talking to; instead you just repeatedly say "here's a chunk of data, send it to X". It's not really relevant that the typical thing to do with these packets is to build a reliable stream on top of them, what matters is that everything except the endpoints can be a lot dumber.
There's something in-between traditional (IP style) packets and traditional (TDM / ISDN style) circuit switching, where you use packets with no bandwidth guarantees, but address them by connection number, not endpoint address.
This still requires you to set up a connection beforehand, but doesn't require you to reserve resources you might not be using.
And while certain telcos had packet network focused on that (Bell and Datakit, for example), and generally OSI cared about including QoS indicators, CLNS and its specific on-the-wire implementation CLNP is so similar to IP that IPv9 explicitly called it out in the proposal to replace IPv4 with it (because of ISO 160bit addressing)
It's also a reflection of the state-of-the-art at the time. Binary protocols were an unmitigated disaster, with standards bodies thinking that applying to the ISO for your organizational ID is a perfectly fine step that anyone does anyway.
That's why there was an entire section for unassigned numbers.
Binary protocols just meant you actually needed to implement serialiser/deserialiser and similar tooling instead of writing dumbest possible riff on strtok() and hoping your software won't be used anymore once DoD internet becomes mature
It’s not that I cannot appreciate the improvements in the space, I’m just wondering if there might be a big part of the design space for widely used protocols that ends up unexplored because the default for almost anything now is HTTP. It has basically become OSI layer 8 at this point.
MIME/Content-Negotiation is essentially OSI Presentation Layer
HTTP sorta acts as stump of ROSE with bit of ACSE. In addition it provides a bit of basic layer for passing some extra attributes that might be considered in-band or out (or side?) band to the actual exchange.
HTTP/3 replaces parts of OSI layer 4 (TCP/UDP). There's a perspective that HTTP has been trying to lower its layer for decades. Especially the way that TLS originated as HTTPS but now is almost a "universal" vertical chunk of Layers 4, 5, and 6. In that perspective HTTP is probably the new Layer 6.
HTTP/3 uses QUIC as a layer 4 equivalent (which is actually just UDP with extra implications in practice). Not many services leverage standards-compliant QUIC other than HTTP/3, but you don't need to do HTTP to get the same protocol working.
If anything, HTTP/3 running on top of QUIC forced shitty middlebox vendors to de-ossify by permitting any QUIC-based protocol, as they cannot practically distinguish a new HTTP/3 connection from a QUIC connection.
Ehh, not really. OSI layer 5 was responsible for managing multiple non-overlapping sessions within a single transport stream, and routing those sessions to specific applications. This is precisely what HTTP/1.1 did (though the accept, content-type, accept-encoding, and transfer-encoding headers are really an implementation of layer 6); QUIC, on the other hand, covers the same layers that TCP does (3-5) plus the aforementioned layer 6.
I recommend actually reading X.200 (the specification of the OSI model) at some point: it's quite approachable (especially for an ITU spec, which are notoriously dense reading), and will quickly make you realize how silly it is that we still use it as a reference for modern stacks.
Another point is that the use of HTTP for everything, outside of the issue of middle boxes breaking protocols for everyone, is that it's essentially capitulation to the wisdom of OSI multi-layered protocols - we replicate their feature sets by reusing bits and pieces of HTTP spec all the time.
> Notoriously so, it's why MIME types and MIME encodings get so complicated.
I made up ULFI because I thought MIME has some problems.
> JSON may not look efficient
Efficiency is not the only issue; there is also the consideration of e.g. what data types you want to use. JSON does not have a proper integer type, does not have a proper binary data type (you must encode it as hex or base64 instead), and is limited about what character sets can be used.
(Also, like other text formats, escaping will be needed.)
> I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet"
I think it can be, depending on the specific use; sometimes it isn't, and will make it worse. (HTTP does have the advantage of having URLs and virtual hosting, although I think it adds too much complexity more than should be needed.) However, I still think that DER is generally better than JSON.
> HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of.
I think it depends on the specific application. However, even then, I think there are better ways than using HTTP with the complexity that it involves, most of which should not be necessary (even though a few parts are helpful, such as virtual hosting).
What are the drawbacks to using the JavaScript Number (really a double float I think) datatype as an integer in an object representation language such as JSON? I've never seen a use case where e.g. 42 (int) could be confused with 42.0 (float). If your application needs specifically an int or a float, then the ingesting application knows that.
If the answer is monetary values, then those should never be floats, and should not be represented in JSON as such. E.g. a dollar and a half should be represented as 150 cents. This follows even for sub-cent precision.
Even if we assume that JSON numbers are JavaScript numbers. There is the problem that some large natural numbers cannot be represented in double or float although some even larger numbers can be represented. This is very bad if you use these numbers as IDs.
scala> (Long.MaxValue-1)
val res4: Long = 9223372036854775806
scala> (Long.MaxValue-1).toDouble.toLong
val res5: Long = 9223372036854775807
The fact that I used Scala is irrelevant here. That is true for many programming languages that 64 bit long and double types.
While the grammar is specified (that’s what JSON is, after all), the runtime representation is unspecified. A conformant JSON parser can parse “1” as 1.0. They can be backed by doubles, or singles, or arbitrary precision.
I don't mean 32-bit integers, but rather, 64-bit (and sometimes bigger) integers. JSON already works OK for 32-bit integers (although it is not ideal).
Much easier would just be native decimal support in JS and we can represent a decimal using d as the decimal point, for example 15d0 (15.0) or 384d25 (384.25).
Using cents instead of dollars sounds fine until you have to do math like VAT, you really need decimal math for that.
> Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet.
That's just not true. Telnet and SMTP are built on top of TCP. They live on the same layer. They were originally both protocols that transmitted data with printable ascii, hence why they look similar. There are many other protocols like Telnet and SMTP that worked like that, auch as nntp, irc, and yes, even http.
I think that the point is this: You could literally use a telnet client to talk to all of those servers and it was considered a good thing, maybe even an essential feature. So protocol design was somehow influenced by the need to be fully plain text ASCII streams (and maybe that other options weren't adequately explored due to this restriction)
People think serializers are easy, when suddenly this makes a whole raft of choices your problem. You can write your own, and now you have two problems; or you can build on top of capnproto or grpc or, if you really want to get retro, ASN.1 and inherit the problems of your serializer framework.
JSON by comparison has simple, obvious limitations that more people are familiar with dealing with.
There's also the tendency to tie your protocol to implementation. The Microsoft Exchange "protocol" didn't get reverse engineered for so long because it's basically the COM structure of Outlook fed through (if I remember rightly) DCOM-RPC.
> I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP
It shouldn't. For some cases it helps, but other times it doesn't. Sometimes it helps but there would be better ways to do it, making it on a simpler protocol or making an entirely new protocol (which might or might not use TCP; sometimes it is better to use TCP and sometimes not) depending on the specific case.
> Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO
I dislike JSON. I think it has many problems, and that DER is a better format.
(There are also the "small web" protocols such as Gemini and Scorpion and Spartan and Titan, which avoids some of the complexity of HTTP; I had considered using DER-over-Scorpion rather than JSON-over-HTTP. It is also possible to use SSH, although SSH does not have virtual hosting.)
How much overhead do you think fetching emails over http takes? Fetching text documents with HTTP seems like the prime usecase for it. And it's the only way you could have something that works in web clients.
I'm struggling to think of any real benefits to not using HTTP other than it would be more interesting.
> Is there really any new-ish binary protocol these days?
HTTP/2 and HTTP/3 are binary protocols. And if you replace the JSON with CBOR, then even the payload becomes binary.
The reason for using HTTP is that the semantics are right. HTTP is a state transfer protocol, and ultimately, that's 90% of what you need for sync.
The other 10% is for subscriptions, updates, with versioning, and patches. You can get these by adding the Braid extensions (see braid.org) which upgrade HTTP from a state transfer to a state synchronization protocol. (I work on Braid.)
JMAP is only JSON over HTTP because that's what all the libraries support. Any data format which provides hashes and arrays would work fine, over any transport. So you could use CBOR or protocol buffers or whatever, over any channel you like.
My hot take is that if the web had been based on binary protocols rather than HTML and JSON, the fragmentation of tooling that can instead be standardized as view-source and network-tabs would have decimated (as in removed 90% of) the hackers that make up this community, and software engineering in general.
There's no magic. Nothing sacred. Nothing that you aren't allowed to understand, intuitively. Nothing where you aren't allowed to imagine "what if it also had X?" The web is yours. The computer is yours. As an industry, we burn some incremental percentage of bandwidth to give you the keys to the kingdom, and to allow you, new developer, to be one of us.
In an age when LLMs feel like magic boxes to tech-minded people new to development, we need this more than ever.
I really hope Fastmail implements the JMAP spec for calendars and contacts soon. They’ve had the mail part of the spec implemented for a while, but it still requires CardDAV/CalDAV for contacts and calendar access.
We're working on it. We still use an unholy set of earlier versions of JMAP internally for our contacts and calendars; in particular the caldav_sync code is gnarly - I wrote it over 10 years ago when I knew less about calendars than I do now! It's still using an earlier branch of what became the perl Net::CalDAVTalk library interally, even though our frontend API is an almost-up-to-date version of what will become the JMAP Calendars spec eventually.
The downsides of developing and testing this stuff as we were writing it up!
We've finished rewriting the objectid generation to give smaller sized and more sortable IDs (they're an inverse of nanosecond internaldates now, plus some extra magic on the low bits for IMAP appends to avoid clashes)... which we wanted to speed up and reduce disk usage for the offline mode.
Next up is indeed updating to the latest spec on calendars and contacts. Files might take a bit longer, I really want to do some desktop clients for the files system, we have a really nice files backend at Fastmail which is only accessible via our interface or WebDAV right now.
Off-Topic: Thank you for not only providing a stellar service with Fastmail, but also for contributing back to the OSS ecosystem and the specifications/RFC work. This takes a lot of time and we all benefit from this work. It helps many people/small IT shops to run a system outside of the "big ones". Again, thank you.
The next, next big thing would be the Chatmail relays[1] supporting JMAP based servers (right now it's Dovecot) and this new targeted push extension for faster notifications without battery drain on mobile. I can see how the Fastmail mobile client will benefit from this RFC as well (it's already incredibly battery efficient, thanks to the team).
What good is a protocol like JMAP as long as common clients like Thunderbird, K-9 Mail, the iPhone email client and others don't support it? Without some concerted effort it will never take off. Then there is the question of what problem it solves that isn't already solved by existing solutions.
The lack of support for JMAP in common mail clients has been my concern for years. And recently Fastmail went ahead with releasing its own Electron-based desktop app that uses JMAP for its mail service. Mozilla Thunderbird could do with some support (financial and/or people wise) to get JMAP implemented. I don’t know if Fastmail the company has done much on this aspect, but hope they do.
I think Thunderbird will implement JMAP at some point, because as I understand it their upcoming hosted mail service, Thunderbird Pro will offer it.
I looked into adding JMAP support to Thunderbird but the client is so tied around the ideas and principles of IMAP, it needs surgical refactoring of many parts of it and I don’t love C++.
So instead in my spare time I am developing a JMAP only gnome email client, using many Stalwart libraries. Think Geary but Rust instead of Vala, GTK4 instead of GTK3 and JMAP instead of IMAP. It’s been mostly an excuse to play with Rust and gtk-rs and Relm4 (beautiful Elm inspired rust bindings for GTK4). Someday, it will be released.
Client support for a new protocol is never that quick, but I believe adoption will happen, at least outside of the big providers, who will never support it.
> We have been experimenting with this for a while now and are using Stalwart as the software stack we are building upon. We have been working with the Stalwart maintainer to improve its capabilities (for instance, we have pushed hard on calendar and contacts being a core piece of the stack).
However, unfortunately I am unsure whether this is a good source or official page.
We need better client support for JMAP. Apple Mail, Thunderbird, Outlook (as if), and so on. I'm surprised some of the smaller ones like Canary or Spark don't implement it as a product differentiator.
One big differentiator is JMAP allows one network connection to track new emails that may get delivered across different folders. With IMAP you need a connection open for each folder.
I'd make a reasonable guess that it enables much better Javascript clients, either via Electron or the Web Browser.
You don't need major providers to support it, they support SMTP and that's how messages are relayed. JMAP is just so you: the client, can fetch your mail from wherever you host your mail.
IMAP is a shit protocol in general. Notably, email UIDs are not guaranteed to be static across sessions (aka, next time you connect they may be different), which is just straight up dumb. You IMAP client has to do a lot of workarounds just for this, which leads to gnarly bugs and inefficiences.
That's not strictly true. IMAP has two things, UIDs which are mostly static (there's a UIDVALIDITY cache invalidation key to let a client know that UID information has been lost and recalculated); and message numbers.
UIDs don't change, but of course they can be deleted so it's a gappy list, meaning you can request even quite a large looking range of UIDs and get nothing back.
Message numbers change in every session, and also change every time you get an EXPUNGE. They're basically an ordered list without gaps, so you do a memmove at the offset of the EXPUNGE each time you get an expunge.
There are efforts like UIDONLY (RFC9586) to avoid having to keep that mapping at all, and there's OBJECTID (RFC8474) to let you cache a lot more even when UIDs are changed or when messages are moved between folders.
I don’t even remotely disagree with this. But that wasn’t the point: the point was that JMAP is only a competitive advantage if (major) email providers actually adopt it. Bad technologies regularly win because of inertia.
I’ve got a friend who’s been pitching me on building a new email client for years. “I’ll do it if we exclusively use JMAP.” “okay does that include Gmail and Apple/iCloud accounts?” “Nope.”
I could sort of see dual-supporting Gmail's proprietary API and JMAP, but unless the #2-5 competitors support it… what’s the point? (sorry to put on the pessimism hat)
There isn’t really a great motivating feature or use-case driving client or server adoption.
To be honest, I’m not sure why end-users would want JMAP for e-mail access.
It would be interesting if they do successfully roll out all of these additional RFC proposals providing a cohesive “groupware” protocol covering calendering, contacts, file shares, etc, we see notable server implementations, and interest is enough to drive client support.
People say things like that, and I wonder if I’ve just been living in a gilded tower of using Apple Mail with decent IMAP server implementations.
I’m also pretty familiar with the wire protocol and its implementation — it’s never struck me as particularly horrible.
A new protocol isn’t likely to solve the problem of poorly implemented clients and servers — e.g. Google doesn’t really care about good IMAP support, so they’re unlikely to care much about JMAP, either. They just want you to use their webapp.
Sounds like it :) I’ve been very happy with Mail.app since MacOS 10.0. My use has always been with my employer’s IMAP servers, and my own cryus (and eventually) dovecot self-hosted IMAP servers.
Mail.app is what NeXT used internally, and Apple uses to this day AFAIK. Steve Jobs historically paid a lot of attention to it and wasn’t shy about weighing in on any changes.
Most of the complaints that I’ve heard about it seemed to stem from poor IMAP servers (e.g. Gmail), but it sounds like your knowledge in the space would be a lot more detailed and recent than mine, so I would be very interested in your thoughts.
Gmail does indeed _intentionally_ provide poor IMAP service. But the long and short of it is that Apple Mail simply isn't a first-class product. It's an afterthought.
It’s such a breeze to self-host your own email server using Stalwart. It has been a new era for email self-hosters like myself since these kind of fully integrated email servers like Stalwart appeared. Another good one but not as actively maintained is Maddy.
I'm setting up Stalwart right now, migrating from my current Maddy+Postfix+Dovecot+Rspamd setup. Not exactly my experience.
The documentation is not great - I'd say it's just about barely enough to get an overall idea, but there's no one proper single definitive overview of what options exist, what are their possible values, what are the defaults, and how they relate to each other. Maddy docs, despite looking a bit sloppy, were a lot easier to get through. IMHO Stalwart makes it unnecessarily difficult to write a non-minimal static configuration file, hooking everything up correctly.
To be fair, maybe there is a page like that but I haven't found it, despite trying.
I know the Web UI allows to do the configuration by clicking through the forms, but this approach conflicts with declarative deployment practices. In my case it's giving me nondescript 500 errors in the UI with "Failed to write local configuration" in the logs because the .toml file is read-only.
Not sure if yours is setup different, but there are several key fields that need to be written to the config.toml file, and I've seen my file get updated when I make changes to the listeners or stores settings.
But in general, I agree that it has not been a very smooth experience. Having messed around with maddy and mox, Stalwart has had quite a few gotchas. Despite being a single binary promising simplicity, I'm finding it to be a real challenge figuring out how it all fits together, and I'm mostly learning by trial and error since the documentation is often outdated.
My biggest gripe is that it doesn't use the config.toml for every setting, or at least doesn't seem to have the option to do so. I broke my installation and had to find the posgresql key-value pairs for the settings, which was made harder by the fact that everything was stored as binary, which also made me have to edit it as binary as well. These were very simple settings that would have been a breeze in a flat configuration file. I absolutely do not like how necessary the WebAdmin is to manage simple things.
That said, the integration with calendar/contacts is nice even without JMAP... Getting Thunderbird and Roundcube setup with plugins and proper settings made it so easy to get several users setup with calendars, contacts, and shared email-boxes and shared contacts right upon first login.
The S3 storage is also working great (Hetzner Frankfurt VPS paired with AWS eu-central-1), and AWS downtime a few days ago notwithstanding, I'm feeling good about the reliability that gives me, leaving me mainly with the PosgresQL data store the main thing to keep backed up.
This is a hugely ambitious software and as such, there will be many things that I will have a hard time getting used to as a hobbyist, but also a lot to be gained. I'm sticking around for now and waiting for version 1, improved documentation, and more clarity on how it all works.
This is awesome to hear, thank you. I have more-or-less figured out how to configure things, and finally found the "local configuration keys" section, but I think I can live with the current setup (mixed toml + db) until Version 1 when I can refactor my configuration. I just installed 0.14 last night and everything went smoothly.
I heard some second-hand accounts that Hetzner's object storage was pretty slow even in the same location. For email, that sounded not too ideal.
Also, I only have 5 mailboxes right now holding less than 15GB of data total... S3 is still cheaper than the minimum at Hetzner since I don't need anything close to a TB.
The nice thing about S3-compatible blob storage is that it is extremely easy to move the objects to another provider (or roll your own MinIO or similar setup) and point your server to it. I initially went with the File-System since it made incremental backups much easier, but ultimately, S3-compatible opened me up to more options and setups.
Yes, are there any decent JMAP web mail client that we can use?
I have asked sooo many times since Stalwart first was introduced, but not got a straight answer. It is just FastMail or Topicbox. I want something like roudcoube or wildduck that can be used over https that I can self-host!
I tried very hard to get it to work, but I simply couldn't get it to connect with my Stalwart instance over JMAP. I do have the permissive CORS and end-points and proxy-protocol seemingly working with my test HTTPS requests, and I also successfully got JMAP to work with the Mailtemi app, but no luck yet with Cypht[0].
Yeah that is kind of issue had me flintch when thinking of using stalwart. As much as it is so nice to install it as a server and ideologies behind it. Looks like gonna just stick with wildduck for now. Just don't like to hedge our email bets on mongodb community edition.
I do have to hand it to the developer though. This is some serious longterm commitment to an open standard that has simply never taken off beyond one company (Fastmail). Current JMAP implementation is pretty much nonexistent, and I am back to using IMAP/WebDAV with Roundcube and plugins with Stalwart. To me, this is an exercise in patience and waiting for an eventual payoff that may or may not come in the next two years. Having followed the project closely for over a year and gone through a few upgrades and followed the community, I'm still optimistic and happy to be along for the ride.
But is there any real benefit over Postfix + Dovecot other than "it's new and written in Rust?" Postfix and Dovecot have been around for decades and respect the Unix philosophy of doing one thing and doing it well.
It's one tiny binary that does everything you could possibly need for hosting a mail server, including an admin UI, and you get a bunch of modern and convenient features for free.
For example, it automatically handles Let's Encrypt certs for you. You get JMAP, CalDAV, WebDAV, CardDAV, IMAP4rev2, DKIM/SPF/DMARC, MTA-STS, DANE, spam filtering, SQL+blob+object storage backends, search, clustering, OpenTelemetry, etc all in one tiny binary.
Downsides: some features are gated behind an enterprise version and I think the dev team is one guy, or at least it was a while ago.
Having ran both for a long time, I'm sticking with Stalwart from now on as long as development continues.
> Downsides: some features are gated behind an enterprise version
I treat this as an insurance policy. Even in this thread people mentioned how Maddy, which is an alternative modern full stack email solution in a single binary, lacks development efforts.
This is why we have this fantastic release for Stalwart - free shit.
Also as of now enterprise is for $0.2 per account per month which is extremely cheap unless somebody wants to build a big spam farm, of which as civilized Internet user I don't support. Obviously this might change, but even if you can always built multi-tenancy layer by yourself if you really need it - rest of the codebase is AGPL.
But the messages are accessible by every open standard: POP, IMAP, JMAP. You can also pick your storage backend, your database backend, your full text/search tool or provider, directory backend. It is amazingly versatile.
Between all the options, you can design incremental backups, snapshots, or whatever with 3rd party tools to write a script to backup your mailboxes to be restorable in any other email service or software. I have tested it with rsync, restic, database dumps, mc/aws-cli depending on the backends used, of which I have tried them all, and found it designed to be very straightforward.
The monolithic aspect is a necessary aspect of being built for HA and distributed environments that it is all the more impressive how versatile it is.
Well yeah, they do one thing, but to do email well you need more than one thing. Dovecot is also kinda shit with UTF-8 support amongst other things. There are plenty of reasons to not use that combination.
What next? Replacing Sieve with something cumbersome, but JSON based?
There is no good desktop implementation of MUA with old technologies (IMAP, Sieve), will all this JMAP help?
I don't think so.
What is profit to have good server with new good (assume it is good, I'm not sure, but lets assume) protocols without good client?
IMAP4 is underused by modern clients: it allows to effectively store client configuration on server, nobody implements it on client side. It allows to configure per-folder Sieve scripts, nobody implements it on client side. Nobody implements good Sieve client (with folder name autocomplition and such) even for global script, not to mention per-folder ones. Heck, there is no good Sieve editor! (I know about Sieve client built on Electron, it is not good, it is incomplete and buggy).
Servers are solved problem (sendmail, exim, postfix, dovecot, cyrus). Clients are not, they stagnated at the moment GMail was announced.
Solved problem? Dovecot can't handle UTF-8 to this day. Things like Stalwart do try and change that and actually support non-legacy features. (Though I think Stalwart doesn't do IMAP NOTIFY either.)
Clients on the other hand have actually kinda moved forward, Apple Mail works with IMAP servers and offers features that people only got with Gmail before. But there are many other examples as well.
What do you mean, dovecot can't handle UTF-8? It works with any properly MIME-encoded message on one hand and allows (even requires) proper UTF-8 for folder and namespace names on the other hand.
I don't have any problems to use UTF-8 in my messages received via dovecot and I have some folders with national characters on my account, and it works both with IMAP and Sieve.
> What is profit to have good server with new good (assume it is good, I'm not sure, but lets assume) protocols without good client?
You need both. You could say, what profit is a good client without a server? By that reasoning, we never stake a step forward without a complete solution.
Now a better mail implementation is just a client away.
Problem with clients not protocols but UI&UX. New protocols can change nothing in this area.
Maybe, Apple Mail is good, again, I don't know as I'm not using Apple, but I don't know any client for Windows and/or OSS with Windows support which has as basic features as support of per-folder settings (identity, sorting mode, etc.) stored on server or proper support for Sieve. In my eyes it is basic features.
And don't let me start to rant about message editor itself, especially in text-only (as opposite to HTML) mode with proper quoting & such.
On a tangent: For do you use format-flowed in text-only that you send? I'm never quite sure if it's universally and effectively supported enough to implement for less-technical users, or better to just define the EOL (and lose dynamic adjustment to viewer width).
I tried to set up stalwart, but I didn't understand exactly what it wanted me to do. It's a webserver (for webmail and admin) and mail server in one, but I already run a webserver and I already have cert infra. So I couldn't figure out what dns settings to use and what ports to reverse proxy. And how to get it to play nice and share the certificate. Seems like stalwart has been designed as if it is the only thing running on a machine, with sparse documentation for any other setup. I tried getting help on the discord server but the tone seemed to be of a sort of "it's quite obvious, you should already know this". At that point, it's so much friction to reverse engineer it that it might just be easier to set up dovecot and postfix.
I have had similarly been unable to get as much help from their Discord as I had hoped.
That said, I'd give this another shot. What I discovered is that Stalwart is incredibly flexible and designed to be used in any configuration you want, and the documentation and examples are incredible, compared to most other projects. BTW, there is no webmail yet, but you are right that there is the admin that is served over HTTP/S, as well as MTA_STS, JMAP. The default listeners get you started and it's kind of out-of-the-scope of Stalwart in terms how you want your server to interact with those listeners. But whether you containerize it or setup a reverse proxy, the documentation has a lot of examples, including how to pick up the certificates, if not managed by Stalwart itself.
I landed on a Caddy for HTTPS and haproxy for proxy protocol on just JMAP, and the remaining TCP services directly binded to the host for the main mail services.
Traefik is another example that I had initially prepped and got working that also handled TCP connections with proxy protocol where needed. The Stalwart documentation was very helpful with that.
In short, I was impressed with how easy it was to integrate Stalwart into whatever setup I wanted, and how open the developer is to different setups. For instance, mox (while awesome in its own right) considers containerized setups to be not-recommended, and generally expects it to live on its own server. Stalwart is flexible, but the side effect is that it is overwhelming at first... but becomes quite elegant once you get the hang of it.
The installation instructions seem to agree with you:
1. curl this shell script
2. Run it as root.
Anytime you see that, you can assume the software wants to take over the whole box and isn't likely to make any attempt to play nicely with other services/software/users already running on the same host.
I feel like the developer is just as serious about containerized setup as letting it take over the whole box. I've run it both ways, and have found it works incredibly well in Docker and the documentation is very good at treating all setups equally.
For those needing to deal with customers/clients/internal teams with Google Workspace/Outlook and wanting JMAP-style (though not JMAP) modern JSON APIs, Nylas might be a viable option: https://www.nylas.com/
Nylas pricing has gotten better recently, but is still quite high though - at $1.50/connected account/month at scale, it's likely material to your per-user margin if it's part of your SaaS offering.
But if you have a use case where this is a no-brainer (like capturing/analyzing/building custom real-time UI around your internal sales team's emails) then it's remarkably powerful.
I worked on the iCalendar, CalDAV, and CardDAV parsers at Apple in 2010 or so, and I see no reason to believe that today’s Macs, iPhones, and Watches, are using anything more modern.
I haven’t been there in more than a decade. I really am curious what the response in Apple (and Google) is to this spec.
Of all of their proposals, this is the most interesting part to me.
I researched what it would take to implement a full calendaring server once, and after reading all the RFCs, just backed away slowly from the whole idea and never thought about it again.
Running Stalwart in production for ~20 heavily used accounts for some company and no problems so far! The simplicity for such a complex stack and flexibility of deployments is off the charts!
Out of curiosity do you front-end SMTP with postfix to have many queues/MX entries and a battle hardened front-end or is Stalwart handling inbound connections directly? Im thinking of moving from Dovecot to Stalwart so family members have more modern features on my fallback domains about half of my domains do not use Fastmail. In multiple companies I had several Postfix inbound servers to keep the internet from touching Exchange directly and have multiple nodes for companies to quickly hand off to in multiple locations.
I just run single instance for now with RocksDB backend for internal / search and S3 for blobs - that is what made me think it’s so flexible.
Never hosted Postfix / Dovecot stack, in fact this is the first time I host emails, but from what I understand Stalwart is designed to handle inbound directly.
For very high throughput inbound you could check out KumaMTA - it was designed specifically for that, but I think Stalwart doesn’t have bottlenecks in it’s clustered topologies which would require it unless you are doing something crazy.
Can you share what's your solution for filtering incoming spam? I've had to abandon Stalwart because its spam filter is so ineffective and inconsistent.
Mind you I am hosting this just for about a week now - +100GB in total for all inboxes. Also I removed automatic daily purging so all spam and deleted items stay just to be safe.
Haven't looked into spam more closely yet. After first glance on most publicly shared email address - there is around 2 spam messages per hour.
Here is report prepared by llm which looked through the last 20 email headers found in spam. All of them were categorized correctly, however there were few emails in the past few days which went to spam where they shouldn't but I think this is fixable.
- Critical Authentication Failures: A large number of the messages failed basic email authentication. We see many instances of SPF_FAIL and VIOLATED_DIRECT_SPF, meaning the sending IP address was not authorized to send emails for that domain. This is a major red flag for spoofing.
- Poor Sender IP Reputation: Many senders were listed on well-known Real-time Blackhole Lists (RBLs). Rules like RBL_SPAMCOP, RBL_MAILSPIKE_VERYBAD, and RBL_VIRUSFREE_BOTNET indicate the sending IPs are known sources of spam or are part of botnets.
- Suspicious Content and Links: The spam filter identified content patterns statistically similar to known spam (BAYES_SPAM) and found links to malicious websites (ABUSE_SURBL, PHISHING).
- Fundamental Technical Misconfigurations: Many sending servers had no Reverse DNS (RDNS_NONE), a common trait of compromised machines used for spam.
There have been few messages which went to spam which didn't meet any of this spam criteria but actually they were cold marketing emails, so it's good too. In addition to this stalwart emits info log for each possible spam message ingested. Not sure if this can get any better than this.
Interesting I didn't know that JMAP for File Storage is a replacement for WebDAV. My experience with WebDAV is that it has traditionally been sluggish and clunky; I'm not sure if that's due to the protocol or the backend. I'm hoping this could be a huge improvement.
It's the latter - podman just sets up all the necessary stuff - namespaces, cgroups, seccomp, network, mounts, etc - and then executes the child. No monitoring whatsoever. The best you can do is to have it listen on the socket it uses for the control API (similar to the docker socket).
However, the quadlets technology allows you to easily setup systemd using systemd generators to initialize the containerized applications using podman and then monitor it for any crashes. Quadlets essentially does everything that docker compose does.
That aside, a container's main overheads aren't the compute or the memory. It's the storage overhead. You're essentially replicating the minimal Linux userland for each container, unless that's in a shared layer.
They make breaking changes to settings (and possibly data stores, but I forget) between versions, so to go from, for example, x.y.1 to x.z.5 might involve doing migrations between x.y.2 through x.z.5 just to use the latest version.
This is not the case for all versions, but I've found it to be common enough that I have to read all of the release notes between point versions when upgrading.
As a JMAP client implementer, there’s a big plus, beside JMAP, in having an integrated server for email, contacts, and files.
Some setups take days even for email, so an easy, fast setup saves a lot of time.
Also, after code the CalDAV ↔ JSCalendar part, using only IANA time zones instead of scattered ones in CalDAV components makes things much simpler.
> They are robust, widely adopted, and battle-tested. Yet, their XML-based design is notoriously verbose, inconsistent, and difficult to implement correctly. Information is scattered across HTTP headers, XML payloads, and even embedded iCalendar data, creating endless compatibility and interoperability challenges between clients and servers.
Can others confirm if these problems are widespread? I get that these protocols are probably a pain to develop for but given they are "robust, widely adopted and battle-tested" it seems that is probably a solved problem. It's better to have one standard that is used everywhere than to have to choose between two standards.
I can understand why JMAP instead of IMAP given the latter's antiquated design. I don't see the advantage to clients in replacing WebDAV though, and the others are a bit iffy too. They'll need to make a way better sales pitch than 'JSON vs XML' (serialization ain't tough, XML is supported everywhere).
I guess contacts/calendar follows JMAP naturally when the clients already implement it, but that only applies in the 'already wrote a JMAP email client' case. Virtually any other case would rather stay with widely supported protocols?
Does it seem like the calendar protocol will be able to replace the VTODO bit of ical so that Todo applications can be built on top of it? I've played around with ics files a bit for the tasks app in nextcloud and it wasn't a pleasant experience so I kinda dropped the project.
> Stalwart Enterprise leverages AI technology to provide unparalleled email security and management. With AI-powered features, Stalwart Enterprise excels in accurately classifying spam, detecting sophisticated phishing attempts, and blocking various types of network attacks. This intelligent approach ensures that your email environment remains secure and reliable. Stalwart Enterprise comes equipped with a pre-trained large language model (LLM), offering robust out-of-the-box protection. Additionally, it supports integration with leading AI providers such as OpenAI, Anthropic, and other cutting-edge platforms, allowing you to enhance and customize your security measures. By utilizing AI, Stalwart Enterprise delivers a smarter, more efficient email solution that proactively safeguards your communications and data.
Anyone got a link to a better sales job on JMAP & friends?
It sounds awesome but the way it is intro'd here:
Over the past few years, the IETF has been redefining how email, calendars, and contacts are synchronized and shared. Building upon the success of JMAP for Mail, several new protocol extensions have been introduced:
JMAP for Calendars - A modern replacement for CalDAV and CalDAV Scheduling.
JMAP for Contacts – A powerful alternative to CardDAV.
JMAP for File Storage – A replacement for WebDAV-based file storage.
JMAP Sharing – A modern successor to WebDAV ACL.
JSCalendar - A clean, JSON-based evolution of iCalendar.
JSContact – A modernized, JSON-native successor to vCard.
...gave me pause. A protocol I've never heard even though I hang out here for an hour a day, was so successful, that it launched 6 new projects?
Sounds more like the parts of the web dev that give me ick (new and shiny; rush to copy new and shiny in other contexts; give it a year; and all of a sudden only 1 of the 6 actually was successful)
The big pitch for JMAP is for a modern web-tech-only approach to email/calendar/"groupware" servers. One reason to do that would be to make it easier to also build email/calendar/"groupware" clients entirely out of modern web-tech. Today most "web email clients" are bespoke to specific stacks/email servers. A dream of JMAP is that with the right CORS policy a single web client could interact with multiple JMAP servers, using only fetch/XHR.
The modernization efforts of JMAP are interesting, too. Most of the old protocols are a mess of bespoke plaintext formats full of quirks evolved over decades in a giant mess of different software. Even the stuff that was already web tech like WebDAV and its extensions CalDAV and CardDAV were full of quirks, violated some REST "rules", and originally intended for a different purpose (file shares/FTP replacement). JMAP is much closer to "plain REST" than WebDAV's complex HTTP protocol extensions/changes.
Counterpoint: I Google'd "jmap gmail" and a top result is a comment from HN in 2019 saying Gmail will never implement JMAP (it has not)
That's a really cruel response, because this is important work. I don't want my kids beholden to bigco.
I think it's real & important.
I also wanna make sure people like me, who have to keep tabs on the intersection of "how can I help liberate from BigCo" and "how can I make a livable wage doing so"
It is, quite literally, real, but also something you shouldn't waste time on if you're already busy. (c.f. https://jmap.io/software.html)
JMAP and friends are very niche, none of the "mainstream" email clients (that ship with most computers/phones) support it. So this feature being available is unlikely to grow the userbase, IMO.
Now JMAP is quite a bit nicer to use than IMAP's API, but IMAP's gravitational field is too strong to be supplanted. IMAP is also becoming somewhat of a niche protocol, as the majority of users use vendor proprietary protocols for accessing their emails on Gmail, Outlook/Hotmail, etc. So why invest the time to add a niche replacement for IMAP when the entire protocol is a second class citizen to mainstream email clients.
If you want to push a new technology, you need to start somewhere. That's exactly what's happening with JMAP. It was created by Fastmail to use as a bridge between their servers and their own apps a case for which popularity doesn't matter. It's basically a modern vendor proprietary protocol but done in the open.
From there, support is only a matter of someone being interested enough to implement it and manifestly it's working. There are now three servers (Apache James, Cyrus and Stalwart) and some clients.
Stalwart is, from what I’ve read, an excellent JMAP server.
JMAP is, from what I’ve read, a great protocol for building an E-Mail (and now also others) client on top of.
Since I would like an innovative way to access my E-Mails, but do not want to self-host, I would find it interesting to use Stalwart as the server component of an E-Mail client: Data is somehow synced into Stalwart via the “ugly” protocols and I get a nice API to build an elegant client on top of.
My basic research shows that something like IMAP-IMAP sync seems to be a thing. Has anyone done something like this, perhaps even with Stalwart? (this of course grows in complexity for each new protocol to be proxied).
I believe having this kind of setup easily accessible could jumpstart a new generation of E-Mail clients on top of JMAP because it (relatively elegantly?) circumvents the chicken-egg problem by allowing all existing IMAP mailboxes to be accessed via JMAP.
I would just like to highlight the "excellent" bit here. It is a tour de force, a masterpiece. Beautifully factored software. They've done an excellent job incrementally building trust and delivering. Really cool project, and I'm so glad it exists.
I also did not realize it is… primarily developed by a single person? The author. Incredible:
https://github.com/stalwartlabs/stalwart/graphs/contributors
This should be pretty straightforward to do with an IMAP <-> IMAP syncing tool, like mbsync [1]. You'd run it periodically in the background to sync the remote IMAP to Stalwart's local IMAP server, and Stalwart can then automatically serve that via JMAP, doing the translation internally.
I was originally thinking you'd need to go remote IMAP <-> maildir <-> Stalwart IMAP, which would be really complicated, but I think the IMAP <-> IMAP should work fine.
[1] https://isync.sourceforge.io/mbsync.html
imapgoose which was recently featured here I think is another option: https://whynothugo.nl/journal/2025/10/15/introducing-imapgoo...
Stalwart currently does not support the NOTIFY extension to IMAP, needed for ImapGoose, though it is planned: https://github.com/stalwartlabs/stalwart/issues/484
ImapGoose is on my radar to replace mbsync and imapnotify in my setup, but I think it's IMAP <-> maildir only, which makes sense for the intended use case of local mail
I have been looking for something like this for ages. Everything I found so far is overpriced, so this is great development.
i have also been pondering this for the exact same reason... unfortunately, i don't have much to show for said pondering... yet...
Reading weirdly many “but there are no clients” posts. Of course something has to come first. You can’t really develop a client without a server implementation and Stalwart is essentially the first server implementation of JMAP.
With Stalwart in place, there’s finally a reason to develop a client for JMAP.
I hope y’all are aware that Mozilla’s new mail service will use it, so that is likely going to give JMAP a big push!
I'm cautiously optimistic that Stalwart is a game-changer. I would normally stick with a simpler platform (like mox) if I wasn't interested in the groupware space. I've previously tried Nextcloud and SoGo and left disappointed, and have more or less been waiting for a project like this to come along.
Nextcloud was such a terrible experience for me (the file sharing/storage was good, but the groupware aspect was incredibly buggy). But knowing that Nextcloud is partnering with Stalwart to hopefully overhaul their stack, Opencloud is developing their JMAP integration, and Mozilla/Thunderbird is using it too (they already have a webmail in development here: https://github.com/thunderbird/stormbox)... we might finally see some exciting development in this space. And now is also a ripe time, as there seems to be a perfect storm of people wanting to get away from Big Tech platforms.
> Stalwart is essentially the first server implementation of JMAP
Just to clarify. Stalwart is the first to have JMAP contacts and calendars. Cyrus has had JMAP mail since the beginning of JMAP.
if I recall correctly FastMail uses JMAP already in production. They were one of the co-authors of the RFC behind it as well
I implemented support for JMAP calendars in Pimsync recently.
You can now synchosise any combination of CalDAV, JMAP, and filesystem.
Stalwart isn't the first implementation of JMAP.
It's the first implementation of JMAP applied to stuff other than Mail.
There are clients. I use one called aerc.
Cyrus was probably the first
While JMAP seems to scratch every itch of a sucker for proper web API design, I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP. Is there really any new-ish binary protocol these days? Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO. Then again, a lot of solid thinking went into these things, so there probably are a lot of good reasons that I’m not aware of.
Still, it’s an interesting space, I think.
> binary protocol
Email was never a binary protocol. Notoriously so, it's why MIME types and MIME encodings get so complicated.
Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet. HTTP as the new telnet has a bunch of improvements when it comes to binary data, request/response-based data flows, and some other considerations. HTTP/3 is even inherently a binary protocol, it's lack of "telnet-compatibility" one of the concerns about switching the majority of the web to it.
vCard/vCal/iCard/iCal were also deeply "plaintext formats". JSON is an improvement because it is more structured, even more efficient, than those predecessors. JSON may not look efficient, but it compresses extremely well and can be quite efficient in gzip and Brotli streams.
I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet", even if it doesn't sound like "binary protocol efficiency" at first glance. Especially as HTTP/3 pushes HTTP more efficient and more "binary", and arguably "more fundamental/basic" with HTTP/3 even taking over more roles in the TCP/UDP layer of the internet stack. (Telnet would never try to replace TCP.) HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of. Sure, it would be neat to see more variety and experiments outside of the HTTP stack, too, but HTTP is too useful at this point not to build a bunch of things on top of it instead of as their own from-scratch protocol.
A lot of the textual nature of older IETF protocols, including the CR LF line endigns, can be probably traced to how easy it was to bang out a bad implementation full of subtle problems that could be debugged by sitting an undergrad student at a teletype instead of spending time on having some binary serializer (that telecom companies definitely had money for)
Yeah, a fair bit of email protocol reeks of "is this tolerant of `telnet mailserver 25` and whatever garbage that might produce".
A lot of old RFCs explicitly mention running on top of TELNET.
Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and one of the earliest ways to access ARPAnet outside of machines directly on it was through calling into a TIP which set up bidirectional stream from your modem to a port on some host.
> Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and
The idea with packets is that you don't need to reserve N bit/s of each link along the route to whatever system you're talking to; instead you just repeatedly say "here's a chunk of data, send it to X". It's not really relevant that the typical thing to do with these packets is to build a reliable stream on top of them, what matters is that everything except the endpoints can be a lot dumber.
There's something in-between traditional (IP style) packets and traditional (TDM / ISDN style) circuit switching, where you use packets with no bandwidth guarantees, but address them by connection number, not endpoint address.
This still requires you to set up a connection beforehand, but doesn't require you to reserve resources you might not be using.
And while certain telcos had packet network focused on that (Bell and Datakit, for example), and generally OSI cared about including QoS indicators, CLNS and its specific on-the-wire implementation CLNP is so similar to IP that IPv9 explicitly called it out in the proposal to replace IPv4 with it (because of ISO 160bit addressing)
It's also a reflection of the state-of-the-art at the time. Binary protocols were an unmitigated disaster, with standards bodies thinking that applying to the ISO for your organizational ID is a perfectly fine step that anyone does anyway.
That's why there was an entire section for unassigned numbers.
Binary protocols just meant you actually needed to implement serialiser/deserialiser and similar tooling instead of writing dumbest possible riff on strtok() and hoping your software won't be used anymore once DoD internet becomes mature
> That's why there was an entire section for unassigned numbers.
That's also why the majority of OIDs in SNMP are rooted in the 1.3.6 hierarchy, which belongs to the DoD.
SNMP OIDs are in 1.3.6 because TCP/IP stack was once more well known as "DoD Internet".
And SNMP is explicitly a DoD Internet simplified alternative to CMIS
It’s not that I cannot appreciate the improvements in the space, I’m just wondering if there might be a big part of the design space for widely used protocols that ends up unexplored because the default for almost anything now is HTTP. It has basically become OSI layer 8 at this point.
MIME/Content-Negotiation is essentially OSI Presentation Layer
HTTP sorta acts as stump of ROSE with bit of ACSE. In addition it provides a bit of basic layer for passing some extra attributes that might be considered in-band or out (or side?) band to the actual exchange.
HTTP/3 replaces parts of OSI layer 4 (TCP/UDP). There's a perspective that HTTP has been trying to lower its layer for decades. Especially the way that TLS originated as HTTPS but now is almost a "universal" vertical chunk of Layers 4, 5, and 6. In that perspective HTTP is probably the new Layer 6.
HTTP/3 uses QUIC as a layer 4 equivalent (which is actually just UDP with extra implications in practice). Not many services leverage standards-compliant QUIC other than HTTP/3, but you don't need to do HTTP to get the same protocol working.
If anything, HTTP/3 running on top of QUIC forced shitty middlebox vendors to de-ossify by permitting any QUIC-based protocol, as they cannot practically distinguish a new HTTP/3 connection from a QUIC connection.
Ehh, not really. OSI layer 5 was responsible for managing multiple non-overlapping sessions within a single transport stream, and routing those sessions to specific applications. This is precisely what HTTP/1.1 did (though the accept, content-type, accept-encoding, and transfer-encoding headers are really an implementation of layer 6); QUIC, on the other hand, covers the same layers that TCP does (3-5) plus the aforementioned layer 6.
I recommend actually reading X.200 (the specification of the OSI model) at some point: it's quite approachable (especially for an ITU spec, which are notoriously dense reading), and will quickly make you realize how silly it is that we still use it as a reference for modern stacks.
Another point is that the use of HTTP for everything, outside of the issue of middle boxes breaking protocols for everyone, is that it's essentially capitulation to the wisdom of OSI multi-layered protocols - we replicate their feature sets by reusing bits and pieces of HTTP spec all the time.
> Notoriously so, it's why MIME types and MIME encodings get so complicated.
I made up ULFI because I thought MIME has some problems.
> JSON may not look efficient
Efficiency is not the only issue; there is also the consideration of e.g. what data types you want to use. JSON does not have a proper integer type, does not have a proper binary data type (you must encode it as hex or base64 instead), and is limited about what character sets can be used.
(Also, like other text formats, escaping will be needed.)
> I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet"
I think it can be, depending on the specific use; sometimes it isn't, and will make it worse. (HTTP does have the advantage of having URLs and virtual hosting, although I think it adds too much complexity more than should be needed.) However, I still think that DER is generally better than JSON.
> HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of.
I think it depends on the specific application. However, even then, I think there are better ways than using HTTP with the complexity that it involves, most of which should not be necessary (even though a few parts are helpful, such as virtual hosting).
If the answer is monetary values, then those should never be floats, and should not be represented in JSON as such. E.g. a dollar and a half should be represented as 150 cents. This follows even for sub-cent precision.
Even if we assume that JSON numbers are JavaScript numbers. There is the problem that some large natural numbers cannot be represented in double or float although some even larger numbers can be represented. This is very bad if you use these numbers as IDs.
The fact that I used Scala is irrelevant here. That is true for many programming languages that 64 bit long and double types.But that problem is not inherent to JSON, which is a pain text encoding of those numbers.
JSON numbers are not JavaScript Numbers.
While the grammar is specified (that’s what JSON is, after all), the runtime representation is unspecified. A conformant JSON parser can parse “1” as 1.0. They can be backed by doubles, or singles, or arbitrary precision.
That's exactly my point. If one needs an integer or a float, the parser converts it. I ask under what use case is there ever ambiguity?
I don't mean 32-bit integers, but rather, 64-bit (and sometimes bigger) integers. JSON already works OK for 32-bit integers (although it is not ideal).
Much easier would just be native decimal support in JS and we can represent a decimal using d as the decimal point, for example 15d0 (15.0) or 384d25 (384.25).
Using cents instead of dollars sounds fine until you have to do math like VAT, you really need decimal math for that.
HTTP is also a large, complex stack for any server/client that isn’t already a web server or a browser.
Yes, this is the other issue with using HTTP.
> Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet.
That's just not true. Telnet and SMTP are built on top of TCP. They live on the same layer. They were originally both protocols that transmitted data with printable ascii, hence why they look similar. There are many other protocols like Telnet and SMTP that worked like that, auch as nntp, irc, and yes, even http.
I think that the point is this: You could literally use a telnet client to talk to all of those servers and it was considered a good thing, maybe even an essential feature. So protocol design was somehow influenced by the need to be fully plain text ASCII streams (and maybe that other options weren't adequately explored due to this restriction)
You can also telnet into port 80 and communicate with an HTTP server.
Unless it is HTTP/3.
or 2.
Or requires HTTPS, for that matter. (But then you can s_client into it :)
Admittedly, it’s been decades since I tried this and I was deliberate about saying http and not https.
Yeah, I got the point. His statements are still factually incorrect and needed correction.
People think serializers are easy, when suddenly this makes a whole raft of choices your problem. You can write your own, and now you have two problems; or you can build on top of capnproto or grpc or, if you really want to get retro, ASN.1 and inherit the problems of your serializer framework.
JSON by comparison has simple, obvious limitations that more people are familiar with dealing with.
There's also the tendency to tie your protocol to implementation. The Microsoft Exchange "protocol" didn't get reverse engineered for so long because it's basically the COM structure of Outlook fed through (if I remember rightly) DCOM-RPC.
> I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP
It shouldn't. For some cases it helps, but other times it doesn't. Sometimes it helps but there would be better ways to do it, making it on a simpler protocol or making an entirely new protocol (which might or might not use TCP; sometimes it is better to use TCP and sometimes not) depending on the specific case.
> Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO
I dislike JSON. I think it has many problems, and that DER is a better format.
(There are also the "small web" protocols such as Gemini and Scorpion and Spartan and Titan, which avoids some of the complexity of HTTP; I had considered using DER-over-Scorpion rather than JSON-over-HTTP. It is also possible to use SSH, although SSH does not have virtual hosting.)
Is der the same as ASN.1?
How much overhead do you think fetching emails over http takes? Fetching text documents with HTTP seems like the prime usecase for it. And it's the only way you could have something that works in web clients.
I'm struggling to think of any real benefits to not using HTTP other than it would be more interesting.
> Is there really any new-ish binary protocol these days?
HTTP/2 and HTTP/3 are binary protocols. And if you replace the JSON with CBOR, then even the payload becomes binary.
The reason for using HTTP is that the semantics are right. HTTP is a state transfer protocol, and ultimately, that's 90% of what you need for sync.
The other 10% is for subscriptions, updates, with versioning, and patches. You can get these by adding the Braid extensions (see braid.org) which upgrade HTTP from a state transfer to a state synchronization protocol. (I work on Braid.)
> I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP.
Absolutely yes, IMO. This significantly eases web client development.
Not being tied to well-known port numbers, and by extension a single service per IP address, is also great.
JMAP is only JSON over HTTP because that's what all the libraries support. Any data format which provides hashes and arrays would work fine, over any transport. So you could use CBOR or protocol buffers or whatever, over any channel you like.
I wonder if you could transparently upgrade to CBOR over HTTP/2.
My hot take is that if the web had been based on binary protocols rather than HTML and JSON, the fragmentation of tooling that can instead be standardized as view-source and network-tabs would have decimated (as in removed 90% of) the hackers that make up this community, and software engineering in general.
There's no magic. Nothing sacred. Nothing that you aren't allowed to understand, intuitively. Nothing where you aren't allowed to imagine "what if it also had X?" The web is yours. The computer is yours. As an industry, we burn some incremental percentage of bandwidth to give you the keys to the kingdom, and to allow you, new developer, to be one of us.
In an age when LLMs feel like magic boxes to tech-minded people new to development, we need this more than ever.
I really hope Fastmail implements the JMAP spec for calendars and contacts soon. They’ve had the mail part of the spec implemented for a while, but it still requires CardDAV/CalDAV for contacts and calendar access.
We're working on it. We still use an unholy set of earlier versions of JMAP internally for our contacts and calendars; in particular the caldav_sync code is gnarly - I wrote it over 10 years ago when I knew less about calendars than I do now! It's still using an earlier branch of what became the perl Net::CalDAVTalk library interally, even though our frontend API is an almost-up-to-date version of what will become the JMAP Calendars spec eventually.
The downsides of developing and testing this stuff as we were writing it up!
We've finished rewriting the objectid generation to give smaller sized and more sortable IDs (they're an inverse of nanosecond internaldates now, plus some extra magic on the low bits for IMAP appends to avoid clashes)... which we wanted to speed up and reduce disk usage for the offline mode.
Next up is indeed updating to the latest spec on calendars and contacts. Files might take a bit longer, I really want to do some desktop clients for the files system, we have a really nice files backend at Fastmail which is only accessible via our interface or WebDAV right now.
Off-Topic: Thank you for not only providing a stellar service with Fastmail, but also for contributing back to the OSS ecosystem and the specifications/RFC work. This takes a lot of time and we all benefit from this work. It helps many people/small IT shops to run a system outside of the "big ones". Again, thank you.
Aside, this bit that Neil and the team are working on is something I think is the 'next big thing': https://www.ietf.org/archive/id/draft-ietf-jmap-emailpush-01...
The next, next big thing would be the Chatmail relays[1] supporting JMAP based servers (right now it's Dovecot) and this new targeted push extension for faster notifications without battery drain on mobile. I can see how the Fastmail mobile client will benefit from this RFC as well (it's already incredibly battery efficient, thanks to the team).
[1] https://github.com/chatmail/relay
Stalwarts supports the latest EmailPush draft too, although it is not announcing it in the JMAP capabilities.
I don’t think JMAP Calendar spec has been ratified yet.
https://datatracker.ietf.org/doc/draft-ietf-jmap-calendars/
And Contacts was only 10-months ago.
https://www.rfc-editor.org/rfc/rfc9610.html
What good is a protocol like JMAP as long as common clients like Thunderbird, K-9 Mail, the iPhone email client and others don't support it? Without some concerted effort it will never take off. Then there is the question of what problem it solves that isn't already solved by existing solutions.
Next time an iCal event invitation screws up the time zone, return to this question.
iCal is timezone-aware, if something goes missing there, that's the fault of a broken implementation.
The lack of support for JMAP in common mail clients has been my concern for years. And recently Fastmail went ahead with releasing its own Electron-based desktop app that uses JMAP for its mail service. Mozilla Thunderbird could do with some support (financial and/or people wise) to get JMAP implemented. I don’t know if Fastmail the company has done much on this aspect, but hope they do.
JMAP works in my email client, aerc.
I think Thunderbird will implement JMAP at some point, because as I understand it their upcoming hosted mail service, Thunderbird Pro will offer it.
I looked into adding JMAP support to Thunderbird but the client is so tied around the ideas and principles of IMAP, it needs surgical refactoring of many parts of it and I don’t love C++.
So instead in my spare time I am developing a JMAP only gnome email client, using many Stalwart libraries. Think Geary but Rust instead of Vala, GTK4 instead of GTK3 and JMAP instead of IMAP. It’s been mostly an excuse to play with Rust and gtk-rs and Relm4 (beautiful Elm inspired rust bindings for GTK4). Someday, it will be released.
Client support for a new protocol is never that quick, but I believe adoption will happen, at least outside of the big providers, who will never support it.
Isn't their upcoming "Thundermail" service actually based on/using Stalwart?
Here is a quote I found on https://thunderbird.topicbox.com/groups/planning/T437cd854af...:
> We have been experimenting with this for a while now and are using Stalwart as the software stack we are building upon. We have been working with the Stalwart maintainer to improve its capabilities (for instance, we have pushed hard on calendar and contacts being a core piece of the stack).
However, unfortunately I am unsure whether this is a good source or official page.
We need better client support for JMAP. Apple Mail, Thunderbird, Outlook (as if), and so on. I'm surprised some of the smaller ones like Canary or Spark don't implement it as a product differentiator.
JMAP is great for thin clients, like webmail, or clients which can’t store any state to disk.
For regular desktop software, I’m not sure that it’s really an improvement over existing protocols.
Serious question: what’s the differentiator if major email providers don’t support it?
(This should not be interpreted as a defense of IMAP.)
One big differentiator is JMAP allows one network connection to track new emails that may get delivered across different folders. With IMAP you need a connection open for each folder.
Implementing IMAP NOTIFY [1] would be a simpler path assuming you have IMAP implemented already though.
[1] https://www.rfc-editor.org/rfc/rfc5465.html
Why is that necessarily a good thing? If there's a problem with one folder - or just a large email - all are delayed?
Okay, that's a great feature! But I guess I'm asking what the differentiator is if major email providers don't use it.
I'd make a reasonable guess that it enables much better Javascript clients, either via Electron or the Web Browser.
You don't need major providers to support it, they support SMTP and that's how messages are relayed. JMAP is just so you: the client, can fetch your mail from wherever you host your mail.
Fastmail supports it, and it sounds like Thundermail will too
Thundermail will be built on Stalwart, so yeah :)
It might be useful to people using non-major email providers.
IMAP is a shit protocol in general. Notably, email UIDs are not guaranteed to be static across sessions (aka, next time you connect they may be different), which is just straight up dumb. You IMAP client has to do a lot of workarounds just for this, which leads to gnarly bugs and inefficiences.
That's not strictly true. IMAP has two things, UIDs which are mostly static (there's a UIDVALIDITY cache invalidation key to let a client know that UID information has been lost and recalculated); and message numbers.
UIDs don't change, but of course they can be deleted so it's a gappy list, meaning you can request even quite a large looking range of UIDs and get nothing back.
Message numbers change in every session, and also change every time you get an EXPUNGE. They're basically an ordered list without gaps, so you do a memmove at the offset of the EXPUNGE each time you get an expunge.
There are efforts like UIDONLY (RFC9586) to avoid having to keep that mapping at all, and there's OBJECTID (RFC8474) to let you cache a lot more even when UIDs are changed or when messages are moved between folders.
I don’t even remotely disagree with this. But that wasn’t the point: the point was that JMAP is only a competitive advantage if (major) email providers actually adopt it. Bad technologies regularly win because of inertia.
> UIDs are not guaranteed to be static across sessions
Realistically speaking, does any server ever rotate its UIDValidity?
We need better server support first.
I’ve got a friend who’s been pitching me on building a new email client for years. “I’ll do it if we exclusively use JMAP.” “okay does that include Gmail and Apple/iCloud accounts?” “Nope.”
I could sort of see dual-supporting Gmail's proprietary API and JMAP, but unless the #2-5 competitors support it… what’s the point? (sorry to put on the pessimism hat)
There isn’t really a great motivating feature or use-case driving client or server adoption.
To be honest, I’m not sure why end-users would want JMAP for e-mail access.
It would be interesting if they do successfully roll out all of these additional RFC proposals providing a cohesive “groupware” protocol covering calendering, contacts, file shares, etc, we see notable server implementations, and interest is enough to drive client support.
That’s a lot of “ifs”.
Because IMAP is horrible, it is another driving reason why we are moving towards the dystopian world of webmail.
Horrible how, exactly?
People say things like that, and I wonder if I’ve just been living in a gilded tower of using Apple Mail with decent IMAP server implementations.
I’m also pretty familiar with the wire protocol and its implementation — it’s never struck me as particularly horrible.
A new protocol isn’t likely to solve the problem of poorly implemented clients and servers — e.g. Google doesn’t really care about good IMAP support, so they’re unlikely to care much about JMAP, either. They just want you to use their webapp.
Gilded tower? Are we living in separate universes?
Shameless plug for a client with true offline-first IMAP support:
https://marcoapp.io
Sounds like it :) I’ve been very happy with Mail.app since MacOS 10.0. My use has always been with my employer’s IMAP servers, and my own cryus (and eventually) dovecot self-hosted IMAP servers.
Mail.app is what NeXT used internally, and Apple uses to this day AFAIK. Steve Jobs historically paid a lot of attention to it and wasn’t shy about weighing in on any changes.
Most of the complaints that I’ve heard about it seemed to stem from poor IMAP servers (e.g. Gmail), but it sounds like your knowledge in the space would be a lot more detailed and recent than mine, so I would be very interested in your thoughts.
Well Jobs is long gone. Apple Mail seems like it hasn't been touched much since his days.
I've written about my experience and motivations here:
https://marcoapp.io/blog/marco-an-introduction
Gmail does indeed _intentionally_ provide poor IMAP service. But the long and short of it is that Apple Mail simply isn't a first-class product. It's an afterthought.
It’s such a breeze to self-host your own email server using Stalwart. It has been a new era for email self-hosters like myself since these kind of fully integrated email servers like Stalwart appeared. Another good one but not as actively maintained is Maddy.
I'm setting up Stalwart right now, migrating from my current Maddy+Postfix+Dovecot+Rspamd setup. Not exactly my experience.
The documentation is not great - I'd say it's just about barely enough to get an overall idea, but there's no one proper single definitive overview of what options exist, what are their possible values, what are the defaults, and how they relate to each other. Maddy docs, despite looking a bit sloppy, were a lot easier to get through. IMHO Stalwart makes it unnecessarily difficult to write a non-minimal static configuration file, hooking everything up correctly.
To be fair, maybe there is a page like that but I haven't found it, despite trying.
I know the Web UI allows to do the configuration by clicking through the forms, but this approach conflicts with declarative deployment practices. In my case it's giving me nondescript 500 errors in the UI with "Failed to write local configuration" in the logs because the .toml file is read-only.
Not sure if yours is setup different, but there are several key fields that need to be written to the config.toml file, and I've seen my file get updated when I make changes to the listeners or stores settings.
But in general, I agree that it has not been a very smooth experience. Having messed around with maddy and mox, Stalwart has had quite a few gotchas. Despite being a single binary promising simplicity, I'm finding it to be a real challenge figuring out how it all fits together, and I'm mostly learning by trial and error since the documentation is often outdated.
My biggest gripe is that it doesn't use the config.toml for every setting, or at least doesn't seem to have the option to do so. I broke my installation and had to find the posgresql key-value pairs for the settings, which was made harder by the fact that everything was stored as binary, which also made me have to edit it as binary as well. These were very simple settings that would have been a breeze in a flat configuration file. I absolutely do not like how necessary the WebAdmin is to manage simple things.
That said, the integration with calendar/contacts is nice even without JMAP... Getting Thunderbird and Roundcube setup with plugins and proper settings made it so easy to get several users setup with calendars, contacts, and shared email-boxes and shared contacts right upon first login.
The S3 storage is also working great (Hetzner Frankfurt VPS paired with AWS eu-central-1), and AWS downtime a few days ago notwithstanding, I'm feeling good about the reliability that gives me, leaving me mainly with the PosgresQL data store the main thing to keep backed up.
This is a hugely ambitious software and as such, there will be many things that I will have a hard time getting used to as a hobbyist, but also a lot to be gained. I'm sticking around for now and waiting for version 1, improved documentation, and more clarity on how it all works.
This will get better soon. The next step on the roadmap is to refactor the configuration format and migrate most APIs to JMAP.
This is awesome to hear, thank you. I have more-or-less figured out how to configure things, and finally found the "local configuration keys" section, but I think I can live with the current setup (mixed toml + db) until Version 1 when I can refactor my configuration. I just installed 0.14 last night and everything went smoothly.
Is there a reason you use aws s3 vs the hetzner object storage?
I heard some second-hand accounts that Hetzner's object storage was pretty slow even in the same location. For email, that sounded not too ideal.
Also, I only have 5 mailboxes right now holding less than 15GB of data total... S3 is still cheaper than the minimum at Hetzner since I don't need anything close to a TB.
Good to know, I thought about using this with stalwart, as I moving to hetzner.
The nice thing about S3-compatible blob storage is that it is extremely easy to move the objects to another provider (or roll your own MinIO or similar setup) and point your server to it. I initially went with the File-System since it made incremental backups much easier, but ultimately, S3-compatible opened me up to more options and setups.
Yes, are there any decent JMAP web mail client that we can use?
I have asked sooo many times since Stalwart first was introduced, but not got a straight answer. It is just FastMail or Topicbox. I want something like roudcoube or wildduck that can be used over https that I can self-host!
It looks like Cypht [0] is the most actively maintained JMAP webmail client listed at [1], assuming that works.
I tried very hard to get it to work, but I simply couldn't get it to connect with my Stalwart instance over JMAP. I do have the permissive CORS and end-points and proxy-protocol seemingly working with my test HTTPS requests, and I also successfully got JMAP to work with the Mailtemi app, but no luck yet with Cypht[0].
Darn. This looks like the open issue: https://github.com/cypht-org/cypht/issues/931
Yeah that is kind of issue had me flintch when thinking of using stalwart. As much as it is so nice to install it as a server and ideologies behind it. Looks like gonna just stick with wildduck for now. Just don't like to hedge our email bets on mongodb community edition.
I do have to hand it to the developer though. This is some serious longterm commitment to an open standard that has simply never taken off beyond one company (Fastmail). Current JMAP implementation is pretty much nonexistent, and I am back to using IMAP/WebDAV with Roundcube and plugins with Stalwart. To me, this is an exercise in patience and waiting for an eventual payoff that may or may not come in the next two years. Having followed the project closely for over a year and gone through a few upgrades and followed the community, I'm still optimistic and happy to be along for the ride.
But is there any real benefit over Postfix + Dovecot other than "it's new and written in Rust?" Postfix and Dovecot have been around for decades and respect the Unix philosophy of doing one thing and doing it well.
It's one tiny binary that does everything you could possibly need for hosting a mail server, including an admin UI, and you get a bunch of modern and convenient features for free.
For example, it automatically handles Let's Encrypt certs for you. You get JMAP, CalDAV, WebDAV, CardDAV, IMAP4rev2, DKIM/SPF/DMARC, MTA-STS, DANE, spam filtering, SQL+blob+object storage backends, search, clustering, OpenTelemetry, etc all in one tiny binary.
Downsides: some features are gated behind an enterprise version and I think the dev team is one guy, or at least it was a while ago.
Having ran both for a long time, I'm sticking with Stalwart from now on as long as development continues.
> Downsides: some features are gated behind an enterprise version
I treat this as an insurance policy. Even in this thread people mentioned how Maddy, which is an alternative modern full stack email solution in a single binary, lacks development efforts.
This is why we have this fantastic release for Stalwart - free shit.
Also as of now enterprise is for $0.2 per account per month which is extremely cheap unless somebody wants to build a big spam farm, of which as civilized Internet user I don't support. Obviously this might change, but even if you can always built multi-tenancy layer by yourself if you really need it - rest of the codebase is AGPL.
What you’re describing sounds like one huge downside to me, not and advantage.
The only way to adopt Stalward is to drop everything else and use a single monolithic do-it-all?
Messages are stored in a bespoke format and not easily accessible directly?
It doesn’t sound like it’s made to be usable with other software. This isn’t an advantage in my book.
But the messages are accessible by every open standard: POP, IMAP, JMAP. You can also pick your storage backend, your database backend, your full text/search tool or provider, directory backend. It is amazingly versatile.
Between all the options, you can design incremental backups, snapshots, or whatever with 3rd party tools to write a script to backup your mailboxes to be restorable in any other email service or software. I have tested it with rsync, restic, database dumps, mc/aws-cli depending on the backends used, of which I have tried them all, and found it designed to be very straightforward.
The monolithic aspect is a necessary aspect of being built for HA and distributed environments that it is all the more impressive how versatile it is.
Well yeah, they do one thing, but to do email well you need more than one thing. Dovecot is also kinda shit with UTF-8 support amongst other things. There are plenty of reasons to not use that combination.
What next? Replacing Sieve with something cumbersome, but JSON based?
There is no good desktop implementation of MUA with old technologies (IMAP, Sieve), will all this JMAP help?
I don't think so.
What is profit to have good server with new good (assume it is good, I'm not sure, but lets assume) protocols without good client?
IMAP4 is underused by modern clients: it allows to effectively store client configuration on server, nobody implements it on client side. It allows to configure per-folder Sieve scripts, nobody implements it on client side. Nobody implements good Sieve client (with folder name autocomplition and such) even for global script, not to mention per-folder ones. Heck, there is no good Sieve editor! (I know about Sieve client built on Electron, it is not good, it is incomplete and buggy).
Servers are solved problem (sendmail, exim, postfix, dovecot, cyrus). Clients are not, they stagnated at the moment GMail was announced.
Solved problem? Dovecot can't handle UTF-8 to this day. Things like Stalwart do try and change that and actually support non-legacy features. (Though I think Stalwart doesn't do IMAP NOTIFY either.)
Clients on the other hand have actually kinda moved forward, Apple Mail works with IMAP servers and offers features that people only got with Gmail before. But there are many other examples as well.
What do you mean, dovecot can't handle UTF-8? It works with any properly MIME-encoded message on one hand and allows (even requires) proper UTF-8 for folder and namespace names on the other hand.
I don't have any problems to use UTF-8 in my messages received via dovecot and I have some folders with national characters on my account, and it works both with IMAP and Sieve.
> What is profit to have good server with new good (assume it is good, I'm not sure, but lets assume) protocols without good client?
You need both. You could say, what profit is a good client without a server? By that reasoning, we never stake a step forward without a complete solution.
Now a better mail implementation is just a client away.
Problem with clients not protocols but UI&UX. New protocols can change nothing in this area.
Maybe, Apple Mail is good, again, I don't know as I'm not using Apple, but I don't know any client for Windows and/or OSS with Windows support which has as basic features as support of per-folder settings (identity, sorting mode, etc.) stored on server or proper support for Sieve. In my eyes it is basic features.
And don't let me start to rant about message editor itself, especially in text-only (as opposite to HTML) mode with proper quoting & such.
On a tangent: For do you use format-flowed in text-only that you send? I'm never quite sure if it's universally and effectively supported enough to implement for less-technical users, or better to just define the EOL (and lose dynamic adjustment to viewer width).
I tried to set up stalwart, but I didn't understand exactly what it wanted me to do. It's a webserver (for webmail and admin) and mail server in one, but I already run a webserver and I already have cert infra. So I couldn't figure out what dns settings to use and what ports to reverse proxy. And how to get it to play nice and share the certificate. Seems like stalwart has been designed as if it is the only thing running on a machine, with sparse documentation for any other setup. I tried getting help on the discord server but the tone seemed to be of a sort of "it's quite obvious, you should already know this". At that point, it's so much friction to reverse engineer it that it might just be easier to set up dovecot and postfix.
I have had similarly been unable to get as much help from their Discord as I had hoped.
That said, I'd give this another shot. What I discovered is that Stalwart is incredibly flexible and designed to be used in any configuration you want, and the documentation and examples are incredible, compared to most other projects. BTW, there is no webmail yet, but you are right that there is the admin that is served over HTTP/S, as well as MTA_STS, JMAP. The default listeners get you started and it's kind of out-of-the-scope of Stalwart in terms how you want your server to interact with those listeners. But whether you containerize it or setup a reverse proxy, the documentation has a lot of examples, including how to pick up the certificates, if not managed by Stalwart itself.
I landed on a Caddy for HTTPS and haproxy for proxy protocol on just JMAP, and the remaining TCP services directly binded to the host for the main mail services.
Traefik is another example that I had initially prepped and got working that also handled TCP connections with proxy protocol where needed. The Stalwart documentation was very helpful with that.
In short, I was impressed with how easy it was to integrate Stalwart into whatever setup I wanted, and how open the developer is to different setups. For instance, mox (while awesome in its own right) considers containerized setups to be not-recommended, and generally expects it to live on its own server. Stalwart is flexible, but the side effect is that it is overwhelming at first... but becomes quite elegant once you get the hang of it.
The installation instructions seem to agree with you:
1. curl this shell script
2. Run it as root.
Anytime you see that, you can assume the software wants to take over the whole box and isn't likely to make any attempt to play nicely with other services/software/users already running on the same host.
To their credit, they do have a docker image (https://stalw.art/docs/install/platform/docker).
I feel like the developer is just as serious about containerized setup as letting it take over the whole box. I've run it both ways, and have found it works incredibly well in Docker and the documentation is very good at treating all setups equally.
For those needing to deal with customers/clients/internal teams with Google Workspace/Outlook and wanting JMAP-style (though not JMAP) modern JSON APIs, Nylas might be a viable option: https://www.nylas.com/
Nylas pricing has gotten better recently, but is still quite high though - at $1.50/connected account/month at scale, it's likely material to your per-user margin if it's part of your SaaS offering.
But if you have a use case where this is a no-brainer (like capturing/analyzing/building custom real-time UI around your internal sales team's emails) then it's remarkably powerful.
I worked on the iCalendar, CalDAV, and CardDAV parsers at Apple in 2010 or so, and I see no reason to believe that today’s Macs, iPhones, and Watches, are using anything more modern.
I haven’t been there in more than a decade. I really am curious what the response in Apple (and Google) is to this spec.
Of all of their proposals, this is the most interesting part to me.
I researched what it would take to implement a full calendaring server once, and after reading all the RFCs, just backed away slowly from the whole idea and never thought about it again.
Yeah. It’s a super messy domain. You never know when the parliament of Brazil will pass a law changing the time zone definition.
How does JMAP solve this?
Running Stalwart in production for ~20 heavily used accounts for some company and no problems so far! The simplicity for such a complex stack and flexibility of deployments is off the charts!
Out of curiosity do you front-end SMTP with postfix to have many queues/MX entries and a battle hardened front-end or is Stalwart handling inbound connections directly? Im thinking of moving from Dovecot to Stalwart so family members have more modern features on my fallback domains about half of my domains do not use Fastmail. In multiple companies I had several Postfix inbound servers to keep the internet from touching Exchange directly and have multiple nodes for companies to quickly hand off to in multiple locations.
I just run single instance for now with RocksDB backend for internal / search and S3 for blobs - that is what made me think it’s so flexible.
Never hosted Postfix / Dovecot stack, in fact this is the first time I host emails, but from what I understand Stalwart is designed to handle inbound directly.
For very high throughput inbound you could check out KumaMTA - it was designed specifically for that, but I think Stalwart doesn’t have bottlenecks in it’s clustered topologies which would require it unless you are doing something crazy.
They have very good docs in general IMO, here are docs on how to cluster - https://stalw.art/docs/cluster/configuration
Can you share what's your solution for filtering incoming spam? I've had to abandon Stalwart because its spam filter is so ineffective and inconsistent.
Mind you I am hosting this just for about a week now - +100GB in total for all inboxes. Also I removed automatic daily purging so all spam and deleted items stay just to be safe.
Haven't looked into spam more closely yet. After first glance on most publicly shared email address - there is around 2 spam messages per hour.
Here is report prepared by llm which looked through the last 20 email headers found in spam. All of them were categorized correctly, however there were few emails in the past few days which went to spam where they shouldn't but I think this is fixable.
- Critical Authentication Failures: A large number of the messages failed basic email authentication. We see many instances of SPF_FAIL and VIOLATED_DIRECT_SPF, meaning the sending IP address was not authorized to send emails for that domain. This is a major red flag for spoofing.
- Poor Sender IP Reputation: Many senders were listed on well-known Real-time Blackhole Lists (RBLs). Rules like RBL_SPAMCOP, RBL_MAILSPIKE_VERYBAD, and RBL_VIRUSFREE_BOTNET indicate the sending IPs are known sources of spam or are part of botnets.
- Suspicious Content and Links: The spam filter identified content patterns statistically similar to known spam (BAYES_SPAM) and found links to malicious websites (ABUSE_SURBL, PHISHING).
- Fundamental Technical Misconfigurations: Many sending servers had no Reverse DNS (RDNS_NONE), a common trait of compromised machines used for spam.
There have been few messages which went to spam which didn't meet any of this spam criteria but actually they were cold marketing emails, so it's good too. In addition to this stalwart emits info log for each possible spam message ingested. Not sure if this can get any better than this.
I second that; only running it for personal use on a few domains, but handles all the complexity _extremely_ easily.
Interesting I didn't know that JMAP for File Storage is a replacement for WebDAV. My experience with WebDAV is that it has traditionally been sluggish and clunky; I'm not sure if that's due to the protocol or the backend. I'm hoping this could be a huge improvement.
I wish there was an easy auto-update process for Stalwart. is anyone hosting an apt repo for it?
edit: we use it on very resource constrained environments, the container version is too much overhead.
Where is the overhead in a container? It is just a regular process. (Ok plus a container runtime process, but that is negligible)
Does podman have a container runtime process? Or does it just exec the child after setting up the environment?
In that case the overhead is just a small amount of kernel accounting.
It's the latter - podman just sets up all the necessary stuff - namespaces, cgroups, seccomp, network, mounts, etc - and then executes the child. No monitoring whatsoever. The best you can do is to have it listen on the socket it uses for the control API (similar to the docker socket).
However, the quadlets technology allows you to easily setup systemd using systemd generators to initialize the containerized applications using podman and then monitor it for any crashes. Quadlets essentially does everything that docker compose does.
That aside, a container's main overheads aren't the compute or the memory. It's the storage overhead. You're essentially replicating the minimal Linux userland for each container, unless that's in a shared layer.
negligible for you, perhaps ;)
isn't it a static binary? Can't you do it the old-school sysadmin way and pull down a binary from github releases and update a symbolic link?
They make breaking changes to settings (and possibly data stores, but I forget) between versions, so to go from, for example, x.y.1 to x.z.5 might involve doing migrations between x.y.2 through x.z.5 just to use the latest version.
This is not the case for all versions, but I've found it to be common enough that I have to read all of the release notes between point versions when upgrading.
It can definitely be improved.
There is a new section in the documentation discussing upgrades:
https://stalw.art/docs/install/upgrade/
The goal is to stabilize the database layout/configuration format very soon so v1.0.0 can be released (hopefully before Q1/Q2 2026).
yes, but that’s not as simple as apt automatic upgrades
want me to write the script out for you?
1. systemd timer
2. curl github api
3. if new release, fetch, verify checksum
4. update symlink
5. restart service
i don’t think repackaging is actually easier here, for main services of a system is ok to skip the package manager.
You will eventually break your email setup by doing this, see my other comment.
What resource constraints allow you to apt-get but not docker pull? It’s the same resulting stack isn’t it?
I am most curious
I didn't realize JMAP had a file system protocol. I'd be very interested to learn how it compares to Solid.
As a JMAP client implementer, there’s a big plus, beside JMAP, in having an integrated server for email, contacts, and files. Some setups take days even for email, so an easy, fast setup saves a lot of time.
Also, after code the CalDAV ↔ JSCalendar part, using only IANA time zones instead of scattered ones in CalDAV components makes things much simpler.
> They are robust, widely adopted, and battle-tested. Yet, their XML-based design is notoriously verbose, inconsistent, and difficult to implement correctly. Information is scattered across HTTP headers, XML payloads, and even embedded iCalendar data, creating endless compatibility and interoperability challenges between clients and servers.
Can others confirm if these problems are widespread? I get that these protocols are probably a pain to develop for but given they are "robust, widely adopted and battle-tested" it seems that is probably a solved problem. It's better to have one standard that is used everywhere than to have to choose between two standards.
You're right to ask it
Always relevant: https://xkcd.com/927/
One of the handful of xkcd numbers I recognize without having to open.
You could say that it's a standard.
I can understand why JMAP instead of IMAP given the latter's antiquated design. I don't see the advantage to clients in replacing WebDAV though, and the others are a bit iffy too. They'll need to make a way better sales pitch than 'JSON vs XML' (serialization ain't tough, XML is supported everywhere).
I guess contacts/calendar follows JMAP naturally when the clients already implement it, but that only applies in the 'already wrote a JMAP email client' case. Virtually any other case would rather stay with widely supported protocols?
> I can understand why JMAP instead of IMAP given the latter's antiquated design.
I think we're about ten years past the point where "newer = better" was a reasonable starting presumption.
Yeah because everything already supports WebDAV. It works well with iOS and Android which is imo a big advantage.
However, doesn't stalwart already also support WebDAV though?
Does it seem like the calendar protocol will be able to replace the VTODO bit of ical so that Todo applications can be built on top of it? I've played around with ics files a bit for the tasks app in nextcloud and it wasn't a pleasant experience so I kinda dropped the project.
if they pull out the AI stuff that'll be soooo cool :D
What AI stuff are you referring to? I just learned about this project from this blog post, so I don't have the full context on their AI work.
From the site [0]:
> Stalwart Enterprise leverages AI technology to provide unparalleled email security and management. With AI-powered features, Stalwart Enterprise excels in accurately classifying spam, detecting sophisticated phishing attempts, and blocking various types of network attacks. This intelligent approach ensures that your email environment remains secure and reliable. Stalwart Enterprise comes equipped with a pre-trained large language model (LLM), offering robust out-of-the-box protection. Additionally, it supports integration with leading AI providers such as OpenAI, Anthropic, and other cutting-edge platforms, allowing you to enhance and customize your security measures. By utilizing AI, Stalwart Enterprise delivers a smarter, more efficient email solution that proactively safeguards your communications and data.
[0]: https://stalw.art/enterprise/
It seems the enterprise edition has AI features and the community version doesn't. So if you don't want AI, use the community version.
https://stalw.art/compare/
Why does the optional (supported only in the enterprise version) feature bother you?
Do you mean the spam detection algorythm or something else?
If the Mimestream mail app supported JMAP, I'd be switching away from Gmail asap.
Anyone got a link to a better sales job on JMAP & friends?
It sounds awesome but the way it is intro'd here:
...gave me pause. A protocol I've never heard even though I hang out here for an hour a day, was so successful, that it launched 6 new projects?Sounds more like the parts of the web dev that give me ick (new and shiny; rush to copy new and shiny in other contexts; give it a year; and all of a sudden only 1 of the 6 actually was successful)
The big pitch for JMAP is for a modern web-tech-only approach to email/calendar/"groupware" servers. One reason to do that would be to make it easier to also build email/calendar/"groupware" clients entirely out of modern web-tech. Today most "web email clients" are bespoke to specific stacks/email servers. A dream of JMAP is that with the right CORS policy a single web client could interact with multiple JMAP servers, using only fetch/XHR.
The modernization efforts of JMAP are interesting, too. Most of the old protocols are a mess of bespoke plaintext formats full of quirks evolved over decades in a giant mess of different software. Even the stuff that was already web tech like WebDAV and its extensions CalDAV and CardDAV were full of quirks, violated some REST "rules", and originally intended for a different purpose (file shares/FTP replacement). JMAP is much closer to "plain REST" than WebDAV's complex HTTP protocol extensions/changes.
You may only just have heard of them, but the WG goes back to 2017.
https://datatracker.ietf.org/wg/jmap/history/
Bron is the principal of fastmail, who now own pobox. This is a serious activity.
Counterpoint: I Google'd "jmap gmail" and a top result is a comment from HN in 2019 saying Gmail will never implement JMAP (it has not)
That's a really cruel response, because this is important work. I don't want my kids beholden to bigco.
I think it's real & important.
I also wanna make sure people like me, who have to keep tabs on the intersection of "how can I help liberate from BigCo" and "how can I make a livable wage doing so"
It is, quite literally, real, but also something you shouldn't waste time on if you're already busy. (c.f. https://jmap.io/software.html)
JMAP and friends are very niche, none of the "mainstream" email clients (that ship with most computers/phones) support it. So this feature being available is unlikely to grow the userbase, IMO.
Now JMAP is quite a bit nicer to use than IMAP's API, but IMAP's gravitational field is too strong to be supplanted. IMAP is also becoming somewhat of a niche protocol, as the majority of users use vendor proprietary protocols for accessing their emails on Gmail, Outlook/Hotmail, etc. So why invest the time to add a niche replacement for IMAP when the entire protocol is a second class citizen to mainstream email clients.
This is a circular self-defeating reasoning.
If you want to push a new technology, you need to start somewhere. That's exactly what's happening with JMAP. It was created by Fastmail to use as a bridge between their servers and their own apps a case for which popularity doesn't matter. It's basically a modern vendor proprietary protocol but done in the open.
From there, support is only a matter of someone being interested enough to implement it and manifestly it's working. There are now three servers (Apache James, Cyrus and Stalwart) and some clients.
If you look it up, you'll see that JMAP is 6 years old now. It's a protocol for doing email (and now other things) over HTTP, without many of the legacy issues from IMAP and SMTP. https://en.wikipedia.org/wiki/JSON_Meta_Application_Protocol / https://jmap.io/index.html
Agreed, also not clear what this or why it matters. This is a new self-hostable email server basically?