Last week Matrix had a presence at UbuCon Europe and PyCon Ireland. We gave workshops on using Matrix to create bots, and also a session on installing Synapse (see also: Brendan's entry below!)
Tomorrow I'm off to sunny Manchester with Michael from the Ops team. We'll visit OggCamp, where we'll show off Matrix with a fun demo I previewed on Matrix Live a few weeks ago.
Matrix will ALSO be at:
Last week we set MSC1219 (key backups), MSC2241 (verification over DMs), and MSC2313 (ban lists) for the spec core team to focus on. Those 3 are rolling on into this week as we didn't get a lot of work done last week :)
In other Spec news, Matthew uncovered a stone tablet describing what would in future be known as "MSC2324":
For a workshop anoa and I did at Ubucon Europe last Sunday on how to install Synapse, I worked on a side project that creates a server, attaches a domain name to it and installs Riot and Caddy on it. Attendees can then SSH into it, follow the instructions to install a Matrix homeserver, and use Riot to register an account on it, log in, and join a federated room with all of the other attendees' homeservers in it. We tried it out last week-end for the actual workshop and it worked quite well 🙂 The project is called Install Party, and lives at https://github.com/babolivier/install-party, and if you want to chat about it I've just created #install-party:abolivier.bzh 😉
This week we've been thinking about the future and brainstorming on ideas to improve perf for small instances and sparing some cycles for MSC1228. Next week we'll return to improving IO usage on matrix.org.
Synapse 1.4.1 was released, which fixes a small regression in 1.4.0.
Just FYI, the matrix slack bridge is now available via NixOS: https://github.com/NixOS/nixpkgs/pull/70753
Remember https://riots.im from last week? Now available on IPFS!
toml's riots.im is also available via IPFS:
ipfs get riots.imor visit
localhost:8080/ipns/riots.imwith ipfs installed and running.
RiotX: we are merging waiting PRs and have postponed the release to ensure a maximum of stability and polishing. The release will be done at the beginning of next week and will contains: read marker, camera picker and improved file picker, share to RiotX capability and many bugfixes.
Heya, matrix-3ds-sdk is a new matrix sdk - for the Nintendo 3DS! It is still deep in development, so expect bugs, some things not working properly etc. Current features include:
- logging in
- sending text messages to rooms
- redacting events from rooms
- resolving aliases to room ids
- basic framework to get a /sync loop working
If you are into 3DS homebrew development or are interested in helping out making a full client based on this, please contact soru (@sorunome:sorunome.de)! 3ds homebrew dev is pretty new to her, so there are plenty of open questions / debugging help would be great!
Support room: #matrix-3ds-sdk:sorunome.de Donate: https://liberapay.com/Sorunome
This is really impressive, I'm looking forward to playing with it. (I'm sure someone at the office must own a 3DS!)
koma, Kotlin library for building clients: Added API for retrieving notifications.
I uploaded the synapse 1.4.1 image to mvgorcum/docker-matrix:v1.4.1 within about half an hour after the release
I've made a docker image of the develop branch of synapse, and automated it to build daily. It's on my docker hub as mvgorcum/docker-matrix:develop, based on the avhost dockerfile. Note that there are no checks, the image is simply built from the develop branch of the synapse git repo every night.
just pushed Version 1.4.1 of the Matrix Synapse for Kubernetes packaging. A bit fiddly to do on a phone 🙂
Black Hat told us:
I'm building the Synapse 1.4.1 multi-arch docker image. It will be pushed to Docker Hub in a couple of hours.
A special section this week from the guy with one editor open for everything, tulir:
I haven't done anything on my own projects this week, but I did contribute to a bunch of other projects:
- Updated SmsMatrix to the latest matrix-android-sdk to fix outgoing sms duplication bug (https://github.com/tijder/SmsMatrix/pull/60)
- Fixed Riot web sending reply fallbacks in edited message content (https://github.com/matrix-org/matrix-react-sdk/pull/3551)
- Fixed some things in the Riot web edit html -> markdown parsing (https://github.com/matrix-org/matrix-react-sdk/pull/3552)
- Made Riot web reply rendering much nicer and more compact (https://github.com/matrix-org/matrix-react-sdk/pull/3553)
- Added full emoji picker for reactions to Riot web (https://github.com/matrix-org/matrix-react-sdk/pull/3554)
The first two are already merged (and SmsMatrix even got a new release on f-droid), the html parsing is waiting for code review and the emoji picker and reply rendering are waiting for design review.
Also, I made a read-only status.matrix.org rss feed room since some people wanted one: #matrix.org-status:maunium.net. I don't remember if I TWIMed these before, but #xkcd:maunium.net and #commitstrip:maunium.net are similar read-only rooms, new xkcds and commitstrips are posted there whenever they come out.
See you next week, and be sure to stop by #twim:matrix.org with your updates!
We've released Synapse 1.4.1 as a small but important bugfix to 1.4.0.
This fixes a regression which crept in with our newly implemented "erase redacted data after N days" feature where some APIs would fail when hitting erased redactions - anyone on Synapse 1.4.0 will want to upgrade asap.
The changelog since 1.4.0 follows:
No changes since 1.4.1rc1.
As you may have heard, New Vector, a major entity in the Matrix world, recently completed a series A funding round.
Mozilla have been trialing different IM solutions to replace IRC, including Matrix. This trial ended this week, and we hope to hear the results this month. (C'mon, Matrix!)
Neil told us:
Matrix.org hit some IO problems earlier this week, while largely a problem with our hosting provider, we’re spending a bit of time to make Synapse more resilient if the same thing were to happen again. This will mean the ability to shard the DB (by table) and spread the load so we are not so dependent on high performance from a given db box. Outside of that we’ve been working on the final polishing of the room directory and getting the sqlite -> Postgres port script into better shape.
Black Hat said:
My multi-arch docker image of Synapse has been updated to v1.4.0.
Not much for Dendrite this week as anoa is off at Ubucon 2019. But we had a few valuable bugs reported by the community, and a pressing reminder to get Dendrite's Monolith mode in as part of its CI.
Half-Shot told us:
Hey folks, The slack bridge 1.0.1 release is out https://github.com/matrix-org/matrix-appservice-slack/releases/tag/1.0.1 containing a few bug fixes found since the original release.
I've not got much for this week, but the IRC bridge has been undergoing some serious refactors and changes for a larger release. Should be quite a big one when it lands :)
Update from poljar:
A PR for riot-web has emerged that adds support for search in E2E encrypted rooms. The PR is utilizing Seshat to perform event indexing and search on riot-desktop. While the PR is missing any sort of UI, it is in a usable state.
Manu told us:
phase:1 of privacy work is done. Riot -iOS 0.10.0 will be available soon
Take a trip down memory lane with the Riots of yesteryear at https://riots.im (note the 's'). Hosting every version of Riot Web released on GitHub 😁
Comes with free Wikipedia hole.
Valere has done a release and is doing some maintenance. He has started to work on integration manager
RiotX: We have fixed quite a lot of issues during the stabilization sprint. We are now working on Sprint 4: read marker, report content, mark all room read, etc. François is changing the media/file picker and we will also be able to share elements from other apps to RiotX. We will schedule a release soon (tm) (should have happen this week, but has been delayed due to stabilization)
uhoreg told us:
Igor, a bot framework for Elixir, has had its first release.
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Massive news for the Matrix ecosystem today: New Vector (the startup which the Matrix core team formed to fund development in 2017) has raised an additional $8.5M of funding in order to speed up Riot/Matrix development and expand Matrix hosting via Modular.im!
The new funding comes in the form of a Series-A equity investment in New Vector from three of the top venture capital funds in London. The round is led by Notion - a fund set up by the founders of MessageLabs, who many will know as one of the leaders in secure hosted email services. Notion's long history with email means they immediately clocked the potential of Matrix's mission to build a new open global communication network - after all, Matrix aims to provide a worthy replacement to email (and the phone network, for that matter!). Joining Notion in the round is First Minute - a fund set up by the founders of Lastminute.com (arguably the UK's most famous original dotcom), and Dawn - one of the largest SaaS tech specialist funds in Europe (famous for backing iZettle, Mimecast, Neo4J and many more).
The last funding round in Jan 2017 from Status was instrumental in stabilising the big 1.0 release of Matrix and exiting beta in June; creating the Matrix.org Foundation as a neutral custodian for the standard; stabilising and optimising Synapse; redesigning Riot’s user interface; bringing in a full-time professional UI/UX designer to the team; supporting the huge amount of encryption work required to turn on E2EE by default (cross-signing, key backups, device verification, e2e search, the pantalaimon e2e daemon etc); creating RiotX/Android; and launching the Modular.im hosting platform.
With today’s new funding, the priorities for Matrix will be:
Turning on end-to-end encryption by default for DMs
Much better support for grouping rooms into Communities
More anti-abuse/anti-spam mechanisms
Shrinking Synapse (and/or finishing Dendrite)
Canonical DMs (having one DM per user, and have them feel clearly distinct from ‘rooms’)
...and furthering development on P2P Matrix, so users can have full control of their communications without having to run or trust a server.
On the New Vector side, this funding will support:
A whole new wave of UX improvements to Riot (particularly around onboarding and first time user experience).
Making Modular hosting as polished and powerful as possible.
Creating a whole new set of next-generation Modular integrations.
While New Vector’s contributions to the Matrix ecosystem can’t be ignored, it’s important to remember that the Matrix protocol and specification itself is governed and controlled by the independent and neutral Matrix.org Foundation and its extensive governance processes. We set up the Foundation very deliberately to enforce the protocol's neutrality, formalise the project's mission, goals and values and hold true to them no matter what - specifically to protect the project from conflicts of interest with commercial Matrix endeavours, including New Vector.
That said, New Vector would not be taking money from any investors if they did not believe their goals are aligned with Matrix's. To clarify:
It turns out that these goals are not incompatible if one understands that the potential of the Matrix ecosystem is directly linked to its openness and size (hint: funding sources who didn’t understand this self-selected out ;). By funding Matrix development and helping the open ecosystem and public network grow, New Vector can go provide more Matrix hosting via Modular.im and more Government & Enterprise deployments via Vector.im. Critically, other companies can and do build on top of Matrix too - and frankly the more players there are, the more valuable the network, and the more value to be shared for everyone (including New Vector). This model worked relatively well for the Web, and we believe it'll work for Matrix too.
Update: the best way to gauge the investment in New Vector is to hear it first hand from the investors. Jos from Notion is leading the round, and has a fascinating blog post (written with zero input from Matrix or New Vector folks) to explain where the investors are coming from.
Update 2: More excellend first-hand analysis from Dan at Dawn Capital, who does a really deep dive into how they see Matrix and New Vector. Another must read.
In this case, all of New Vector's new investors have a background as respected tech entrepreneurs, and everyone involved categorically understands that Matrix itself is a neutral open source project, and the mission is to help build up the whole network to be as successful as possible rather than sabotage it by constraining it in any way.
All in all, it’s great news for the ecosystem: Matrix is 5 years old now, and while the project is growing faster than ever (over 300% more active users in the last year!) - it's fair to say that we haven't moved as fast as our mainstream competition - for instance, Slack is only a year older, and Discord is a year younger(!) Obviously much of this is due to Matrix being a completely different proposition: we've been creating an open spec; multiple client codebases; multiple server codebases; the bridges; a fault tolerant decentralised network - not to mention the complexities of decentralised E2E encryption. Based on comparing with our endeavours prior to Matrix, we estimate building this stuff in an open and decentralised manner takes roughly 6 times longer.
But the project is now in a position where the foundations are solid: the protocol is out of beta, reference servers and clients are production ready, and it’s more than time to make all of this mainstream. We have to redouble our focus on user experience and ensure that we compare favourably to today’s established alternatives while staying true to Matrix’s principles. Making sure there are Matrix apps out there which provide a credible alternative to with the likes of Slack and WhatsApp (until they eventually join Matrix, of course) is what will make the difference between Matrix being a cliquey FOSS curiosity versus really being the natural successor to today’s instant messaging, email and phone networks.
In the end; Matrix needs full-time contributors in order to continue to grow, and keeping New Vector funded is a very good way to achieve that (New Vector is hiring!). (That said, if any philanthropic billionaires are reading this, the Matrix.org Foundation is actively soliciting donations to improve Matrix independently of New Vector's efforts - particularly around the areas of countering online abuse and disinformation).
In the meantime, huge thanks to Jos at Notion for believing in Matrix and leading this funding round in New Vector - and huge thanks to the other investors who saw the potential! And most of all, thank you to all those supporting Matrix, whether by donating to the Foundation, promoting and using the protocol, or contributing code to the ecosystem. You are the ones keeping the dream alive :)
You can read things from the NV angle over at https://blog.vector.im/8-5m-to-accelerate-matrix/. We hope you’re as excited as we are to open a whole new chapter as Matrix picks up yet more momentum :D
-- Matthew, Amandine, and the whole Matrix team
Two MSCs have reached final comment period this week:
Last week the spec core team said they'd start focusing on 3 MSCs per week. Those were MSC2290, MSC2176 and MSC1219. The first two have entered FCP, and MSC1219 will roll over into this week.
Thus, the 3 MSCs the spec core team will be focusing on this week will be MSC2199, MSC1219 and an up and coming security-related MSC. Join us in #matrix-spec:matrix.org for related spec discussions :)
There is also a new MSC, MSC2312 describing the proposed Matrix URI scheme. This is a remake of a much older proposal. The general idea is to make a standard for Matrix URIs:
Synapse This week we shipped our privacy release 1.4.0 which is a huge deal in improving user data privacy. Additionally we also included a significant perf improvement which will help anyone suffering from a build up in forward extremities.
Coming up, improved room directory perf, ironing out wrinkles in the room upgrade UX as well as a major reliability boost in the sqlite -> Postgres db porting script.
Dendrite's latest hiatus has come to a close after the privacy work had taken so much of my time. Thankfully although PR review is blocked on the dendrite team, the community have continued to submit PRs and even review each other's PRs (thanks cnly !).
Fixes this week have mostly focused around the CI. We finally got Dendrite's CI unborked (t'was borked in a half-way transition between CircleCI and Buildkite), but it's now working and faster than ever. We're also working to add the Sytest test failure results to the top of the CI window such that people can see which tests failed and the associated Dendrite logs.
Additionally we had some new and merged PRs this week! A federation fix from cnly, a change to allow Dendrite to better work in kubernetes setups by aditsachde, and a codebase fix from @manasseh:matrix.org, who's also working on another fix involving updating gomatrixserverlib to support more recent spec versions.
Some things people were asking about:
- Progress of dendrite is tracked in Dendrite's milestones. We're currently aiming for #1 (Client-Side) Bot Hosting.
- CI not running for community PRs - Buildkite hosts currently run multiple PRs from many different projects, so we can't trust arbitrary code to run on them quite yet. There's a project in progress to run the code in a sandboxed environment (think VM or container) to lift this restriction :)
Aaaaaaaaaaand shipped 🥳 https://github.com/matrix-org/matrix-appservice-slack (1.0 is out!)
Check out the announcement post, big congrats to Half-Shot for getting this out.
Big release for riot-web, check out their release: https://medium.com/@RiotChat/new-privacy-controls-for-riot-dc3661888563
Riot-Android: 2 main things have landed on develop, to be released early next week:
- Catching up on riot-web new Privacy Controls (choose identity server, stun server, securely compare contacts)
- [fdroid only] A new background sync mode for notifications. You can now choose between 'optimized for battery' and 'optimize for realtime'
- build available on buildkite
- Read Markers have landed on develop (jump to last read) Focus on stabilization and bug fixes
Continuum, desktop client written in Kotlin, version 0.9.26:
- Width of columns can be adjusted by dragging, this is a screenshot showing the mouse cursor placed between the first two colomns.
- Fix occasional out-of-bounds array access errors while calculating bounds during layout passes.
As a proof of concept, I have created a little Matrix client with Flutter for Web, named FluffyWeb. This client has only basic features but it shows the possibilities of Flutter for Web and it seems to work fine so far. The client has a responsive design and should work on mobile fine too. Check it out at: https://christianpauly.gitlab.io/fluffyweb/ (Not working in Internet Explorer - I recommend the AOL Messenger in this case)
Just released version 1.4.0 of the Ruby SDK, the main change in this release is the addition of a naïve set of methods to replace the logger implementation. This should allow the gem to slot more easily into projects where existing logging configuration is already in place.
With the release of Synapse 1.4.0, there was a rush to get packages and containers updated, the community are always fast, but we should acknowledge Anance for having his k8s images updated within a few minutes!
matrix-docker-ansible-deploy has also been updated to support Synapse v1.4.0 and riot-web v1.4.0.
As always, referring to the project's changelog before upgrading is a good idea.
KUDOS to this this project! I love that I can
ansible-playbook -i inventory/hosts setup.yml --tags=setup-all without really checking and my toy homeserver is automagically updated.
Just pushed the Kubernetes-optimized images for Synapse version 1.4.0
I set up my own docker-hub account to push images of RC's of synapse at mvgorcum/docker-matrix that aren't built for the avhost/docker-matrix repo
I made a bot that gives the link for msc names. Code is here: https://github.com/anoadragon453/msc-chatbot
Welcome to the JRuby team who announced they will be moving their official chat to Matrix.
Let's reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Mastodon released v3.0.0 - check it out.
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Hello Matrix enthusiasts! Yesterday we released matrix-appservice-slack 1.0. This marks a major milestone in bridge development for the Matrix.org team, being our first bridge to ever reach 1.0. The decision to release this version came after we decided that the bridge had gained enough features and reached a point of stability where it could be deployed in the wild with minimal risk.
For those not in the know, the Slack bridge is Node.JS based, and bridges slack channels & users into Matrix seamlessly. And for those wondering, yes it works with Mattermost too (since their API is compatible with Slack)! In previous versions only a limited subset of features were supported, making heavy usage of Slack’s webhook API. As of 1.0, the bridge now makes use of the newer Slack Events/RTM API which gives us all we need for a richer bridging experience. Everything from edits and reactions to typing notifications is supported in the 1.0 release.
Finally for those who are self hosting, we are pleased to offer the ability to "puppet" your Slack account using the bridge. Puppeting is the process in which the bridge will send messages as if you were sending them from the Slack client directly, when you talk using your Matrix account. This opens the door to seamless bridging and direct messaging support.
Threading & Reactions!
The bridge has undergone some pretty serious code surgery as well. The whole codebase has been rewritten in TypeScript to take advantage of type checking and generic types. The bridge is currently based upon the matrix-appservice-bridge library. The datastore interface now supports PostgreSQL, wich allows for administrators to inspect and edit the database while the bridge is running, as well as offering helpful performance boost over the NeDB datastore format that was used previously. Finally, the codebase has proper Unit and Integration Tests to ensure new changes will not cause any regressions in behaviour. In short, now is an excellent time to get involved and hack on the bridge. There is already a crafted list of easy issues for new and experienced bridge devs.
Memory usage of the bridge comparison
CPU usage of the bridge comparison
In terms of how many users matrix.org is currently serving at the moment, we present to you some figures:
Of course, our work doesn’t stop at 1.0. The plan for the immediate future of the bridge is to continue adding support for other event types coming from Matrix and Slack to create an ever richer experience. Obvious features are things like topic changes, and syncing membership across the bridge. In the long term future we would also like to add community support to the bridge, so whole Slack workspaces can be bridged across with a single click.
That’s all from me, and I would like to say a massive thank you to Cadair and Ben for both their code and review work on the project and as always, thank you to the community for using the bridge and reporting issues. 🙂