This Week in Matrix 2018-12-28

2018-12-28 — This Week in Matrix — Ben Parsons

🔗Welcome to the last TWIM of the year

It's the 28th of December as I write this, and I hope you had a good year. Long as it is, I recommend reading Matthew's write-up of the year, as it covers a lot of ground!

Many of the core team have been out of the office this week, but there are still plenty of updates to share from the Matrix ecosystem!

🔗mxisd v1.2.2

Max released mxisd v1.2.2:

mxisd had a Holiday-special release: v1.2.2 before v1.3.0.

This release introduces two new big features:

Username login rewriting via 3PID to allow advanced flows, like bypassing the synapse restriction of having numerical usernames for non-guest users

Support for multiple Base DNs for LDAP backends

Work has started on v1.3.0 so this is definitely the last release before a non-backward compatible release.

🔗Scylla

VaNilLa has started a new client, a web app build in Elm:

Hi all! I am working on a Matrix client in Elm, and I was recommended to share it here: Scylla

🔗koma-desktop

druig:

koma-desktop is updated to JavaFx 11 and installation is simplified. Dependencies, including native modules can be packaged into one single file, which only needs to be downloaded and run. Java Runtime 11 is the only runtime dependency. Now it's just cross-compilation that needs to be set up before packaged releases can be provided for Mac, Windows, and Linux users.

🔗Prometheus Alertmanager bot for Matrix

Jason Robinson:

I started work on a Prometheus Alertmanager bot for Matrix. The basic idea is that Alertmanager can send webhook alert events to the bot which will then send the formatted events to configured rooms based on the alert receiver. It works, but is still early work in progress. See code and info here: https://git.feneas.org/jaywink/matrix-alertmanager.

Also mirrored on GitHub: https://github.com/jaywink/matrix-alertmanager

🔗Fractal 4.0

Alexandre Franke:

As expected the Fractal team released 4.0 and is already hard at work on the next micro version. We recommend getting it from Flathub like we usually do.

From the release notes:

New features:

Enhanced history view with adaptive layout, day divider

Reorganised headerbar, app menu merged with user menu

Larger display of emoji-only messages

Some performance improvements

Opening a room jumps to first unread message

Bugfixes:

More reliable notifications

Fixed display bug for avatars

Under the hood:

Large code refactor

Logging infrastructure

Continuous integration

More informative build output

🔗maubot

tulir working on maubot:

The new command handling system in maubot is ready. The new system should be much nicer to use when developing plugins.

Previously maubot had a system that was designed after the improved bot support spec proposal, but it wasn't very nice or pythonic. If/when the proposal or something similar goes through, I'll probably add support for it in the new command handling system.

Next I'll make some developer docs so that other people could actually make their own plugins.

The code is at https://github.com/maubot/maubot and you can ask about maubot in #maubot:maunium.net.

🔗matrix-client.el new features

alphapapa:

Emacs makes it so easy to integrate things. Now you can send org-mode syntax messages with the /org command in matrix-client.el.

Tab-completion of usernames and IDs was added to matrix-client.el.

🔗matrix-to GitHub app

t3chguy has come out of hiding to announce:

https://github.com/apps/matrix-to is a Github App which makes use of their shiny and new Content Attachments API/Webhook. When a matrix.to or view.matrix.org URL is used this app is activated. It adds a little snippet with the Room Title and Topic (if the room is peekable from matrix.org). In future it'll work for event permalinks, but currently there is no support for peeking context/event in Matrix API.

Example can be seen at https://github.com/matrix-org/matrix.to/issues/52#issuecomment-449878490. Idea courtesy of TravisR.

🔗See you next year

So there you have it.

I'm at 35c3 with some known characters from the Matrix world (as well as 15,000 others.) If you're here too, come visit us in our assembly, and also make sure to come to Dijkstra tomorrow to watch me present a look back at on the last year: https://fahrplan.events.ccc.de/congress/2018/Fahrplan/events/9400.html. We have recorded a message for Matrix Live from 35c3, but will post tomorrow with some more footage from the event.

Otherwise, see you next year Matrix fan!

The 2018 Matrix Holiday Special!

2018-12-25 — General, Holiday Special — Matthew Hodgson

Hi all,

It's that time again where we break out the mince pies and brandy butter (at least for those of us in the UK) and look back on the year to see how far Matrix has come, as well as anticipate what 2019 may bring!

🔗Overview

It's fair to say that 2018 has been a pretty crazy year. We have had one overriding goal: to take the funding we received in January, stabilise and freeze the protocol and get it and the reference implementations out of beta and to a 1.0 - to provide a genuinely open and decentralised mainstream alternative to the likes of Slack, Discord, WhatsApp etc. What's so crazy about that, you might ask?

Well, in parallel with this we've also seen adoption of Matrix accelerating ahead of our dev plan at an unprecedented speed: with France selecting Matrix to power the communication infrastructure of its whole public sector - first trialling over the summer, and now confirmed for full roll-out as of a few weeks ago. Meanwhile there are several other similar-sized projects on the horizon which we can't talk about yet. We've had the growing pains of establishing New Vector as a startup in order to hire the core team and support these projects. We've launched Modular to provide professional-quality SaaS Matrix hosting for the wider community and help fund the team. And most importantly, we've also been establishing the non-profit Matrix.org Foundation to formalise the open governance of the Matrix protocol and protect and isolate it from any of the for-profit work.

Porting Synapse to Python 3

2018-12-21 — Tech — Neil Johnson

Matrix's reference homeserver, Synapse, is written in Python and uses the Twisted networking framework to power its bitslinging across the Internet. The Python version used has been strictly Python 2.7, the last supported version of Python 2, but as of this week that changes! Since Twisted and our other upstream dependencies now support the newest version of Python, Python 3, we are now able to finish the jump and port Synapse to use it by default. The port has been done in a backwards compatible way, written in a subset of Python that is usable in both Python 2 and Python 3, meaning your existing Synapse installs still work on Python 2, while preparing us for a Python 3 future.

🔗Why port?

Porting Synapse to Python 3 prepares Synapse for a post-Python 2 world, currently scheduled for 2020. After the 1st of January in 2020, Python 2 will no longer be supported by the core Python developers and no bugfixes (even critical security ones) will be issued. As the security of software depends very much on the runtime and libraries it is running on top of, this means that by then all Python 2 software in use should have moved to Python 3 or other runtimes.

The Python 3 port has benefits other than just preparing for the End of Life of Python 2.7. Successive versions of Python 3 have improved the standard library, provided newer and clearer syntax for asynchronous code, added opt-in static typing to reduce bugs, and contained incremental performance and memory management improvements. These features, once Synapse stops supporting Python 2, can then be fully utilised to make Synapse's codebase clearer and more performant. One bonus that we get immediately, though, is Python 3's memory compaction of Unicode strings. Rather than storing as UCS-2/UTF-16 or UCS-4/UTF-32, it will instead store it in the smallest possible representation giving a 50%-75% memory improvement for strings only containing Latin-1 characters, such as nearly all dictionary keys, hashes, IDs, and a large proportion of messages being processed from English speaking countries. Non-English text will also see a memory improvement, as it can be commonly stored in only two bytes instead of the four in a UCS-4 “wide” Python 2 build.

Editor's note: If you were wondering how this fits in with Dendrite (the next-gen golang homeserver): our plan is to use Synapse as the reference homeserver for all the current work going on with landing a 1.0 release of the Matrix spec: it makes no sense to try to iterate and converge on 1.0 on both Synapse and Dendrite in parallel. In order to prove that the 1.0 spec is indeed fit for purpose we then also need Synapse to exit beta and hit a 1.0 too, hence the investment to get it there. It's worth noting that over the last year we've been plugging away solidly improving Synapse in general (especially given the increasing number of high-profile deployments out there), so we're committed to getting Synapse to a formal production grade release and supporting it in the long term. Meanwhile, Dendrite development is still progressing - currently acting as a place to experiment with more radical blue-sky architectural changes, especially in low-footprint or even clientside homeservers. We expect it to catch up with Synapse once 1.0 is out the door; and meanwhile Synapse is increasingly benefiting from performance work inspired by Dendrite.

🔗When will the port be released?

The port is has been released in a “production ready” form in Synapse 0.34.0, supporting Python 3.5, 3.6, and 3.7. This will work on installations with and without workers.

🔗What's it like in the real world?

Beta testers of the Python 3 port have reported lower memory usage, including lower memory “spikes” and slower memory growth. You can see this demonstrated on matrix.org:

See 10/15, ~20:00 for the Python 3 migration. This is on some of the Synchrotrons on matrix.org.

See ~11/8 for the Python 3 migration. This is on the Synapse master on matrix.org.

We have also noticed some better CPU utilisation:

See 21:30 for the migration of federation reader 1, and 21:55 for the others. The federation reader is a particular pathological case, where the replacement of lists with iterators internally on Python 3 has given us some big boosts.

See 10/15, 4:00.The CPU utilisation has gone down on synchrotron 1 after the Python 3 migration, but not as dramatically as the federation reader. Synchrotron 3 was migrated a few days later.

As some extra data-points, my personal HS consumes about 300MB now at initial start, and grows to approximately 800MB -- under Python 2 the growth would be near-immediate to roughly 1.4GB.

🔗Where to from here?

Python 2 is still a supported platform for running Synapse for the time being. We plan on ending mainstream support on 1st April 2019, where upon Python 3.5+ will be the only officially supported platform. Additionally, we will give notice ahead of time once we are ready to remove Python 2.7 compatibility from the codebase (which will be no sooner than 1st April). Although slightly inconvenient, we hope that this gives our users and integrators adequate time to migrate, whilst giving us the flexibility to use modern Python features and make Synapse a better piece of software to help power the Matrix community.

🔗How can I try it?

The port is compatible with existing homeservers and configurations, so if you install Synapse inside a Python 3 virtualenv, you can run it from there. Of course, this differs based on your installation method, operating system, and what version of Python 3 you wish to use. Full upgrade notes live here but if you're having problems or want to discuss specific packagings of Synapse please come ask in #synapse:matrix.org.

🔗Thanks

Many thanks go to fellow Synapse developers Erik and Rich for code review, as well as community contributors such as notafile and krombel for laying the foundations many months ago allowing this port to happen. Without them, this wouldn't have happened.

Happy Matrixing,

Amber Brown (hawkowl)

This Week in Matrix 2018-12-21

2018-12-21 — This Week in Matrix — Ben Parsons

🔗Matrix Live: Half-Shot talks bridges, and working on libpurple bridging to Matrix

You may have seen that Half-Shot been working fearlessly and tirelessly on bridges for many, many months. In this episode of Matrix Live Half-Shot chats with Matthew about his work, progress and achievements, with a focus on recent matrix-appservice-purple and XMPP work. Audio is not amazing, but worth listening to get acquainted with recent work.

As a note from Half-Shot:

The matrix-appservice-purple bridge gained a XMPP specific backend for better performance when you want to do just xmpp bridging. It's rather quick right now, and needs dogfooding.

Chat in #purple-bridge:half-shot.uk!

🔗Matrix badges on shields.io

Brendan:

A couple of weeks ago, fr1kin PR'd a nice Matrix badge to the Shields project that tells you how many users are in a public room. There were a few issues with it so I PR'd some changes to make the badge more usable, and it's now merged and live (as of yesterday evening), with examples available here! ?
For instance, here's a badge for TWIM: https://img.shields.io/matrix/twim:matrix.org.svg ?

🔗Spec update

New MSC for cross-signing, which has different (hopefully better) semantics from the previous cross-signing MSC.

🔗matrix-bot-sdk support for application services

TravisR, author of matrix-bot-sdk:

matrix-bot-sdk has received early support for application services. Similar to the official matrix-js-sdk, the bot-sdk uses an Intent-based model for making bridges easier to write. Check out the simple example here for more information on how it works.

🔗koma-library

druig continues work on their JavaFX/kotlin client project:

The matrix client API implementation in koma is being extracted into a new repo, which is going to be a lightweight library for Kotlin.

🔗Seaglass now available on homebrew

Aaron Raimist reports that Seaglass is available on homebrew for macOS:

Installing Seaglass is now easier than ever. If you already use Homebrew to manage other packages, you can now install Seaglass with a simple brew cask install seaglass.

If you don't use Homebrew, you can still download Seaglass from GitHub.

🔗matrix-client.el is reborn!

alphapapa reports from a team who have forked and are maintaining a Matrix client for Emacs:

Many additions and improvements to matrix-client.el (https://github.com/jgkamat/matrix-client-el) recently, including a "standalone client" mode that launches Emacs to look like this.

Chat in #matrix-client-el:matrix.org.

🔗matrix-docker-ansible-deploy

Slavi:

the matrix-docker-ansible-deploy playbook has received some bugfixes and improvements lately. Most importantly, it's now running the freshly-released Synapse 0.34.0 under Python 3, so memory usage should be much better.

🔗libQMatrixClient/Quaternion

kitsune:

I started work on matrix: scheme support in libQMatrixClient/Quaternion. Expect more news on this around New Year.

🔗Fractal road to 4.0

Fresh from their hackfest in Seville last week, Alexandre Franke reports:

The Fractal crew spent the week chasing last minutes bugs and made two beta releases (3.99.0 and 3.99.1) in preparation for the big new stable release, 4.0, which is right around the corner.

🔗New Rooms for Space Launches and Aviation

Aaron Raimist has been creating some new rooms:

#space:im.kabi.tk for anyone interested in space, rocket launches, satellites, etc.
Are you wondering what NASA's new Mars Rover is doing? Maybe you live on the west coast of the United States, and you saw that meteor on Wednesday night that came within minutes of a scheduled rocket launch and just after three astronauts left the space station. If any of that sounds interesting to you, feel free to join the room.
A Matrix bot is being tested to send updates about upcoming rocket launches to the room.

#aviation:matrix.org for anyone interested in aviation. Whether you are a pilot, someone who visits an airshow once in a while, or if https://www.youtube.com/watch?v=CXv1j3GbgLk piqued your interest, come join.

🔗t2bot.io upgrade

TravisR:

Just a head's up that I've increased the storage capacity of the database. With current projections, the server should be good for another year or two.

The database is also 1ms closer to Synapse and has a faster CPU in it. It probably won't make a dent in speed on federation, but it is a step forward.

I'll be rolling out python 3 to the homeserver this week too, which should help a little bit.

https://riot.im/experimental/#/room/#help:t2bot.io/$154511414912clELm:t2l.io

🔗FluffyChat 8.0 RC announced

Krille and his Ubuntu Touch fans are looking forward to FluffyChat 8.0:

Hey guys, in order to release the FluffyChat 8.0 Christmas Edition, the FluffyChat 8.0 release candidate is ready for you! :-)
Also the weblate translations are ready: https://hosted.weblate.org/projects/fluffychat/

🔗Riot iOS

From the Riot iOS team:

We have fixed and improved some e2e stuff.
Back to reskin. We start to implement e2e backup screens.

🔗Riot Android

From the Riot Android team:

New screen to troubleshoot notification issue has been merged on /develop.
Splitting current Android SDK to separate crypto part is on it's way. We're doing this in order to be able to integrate crypto faster in the Riot reboot.

🔗Synapse

Neil from Synapse reports:

We released 0.34.0! This release recommends Python 3 for production and brings with it huge performance improvements. If you've been putting upgrading off upgrading your Synapse, now is the time to do so. For more details here is a post that explains what you should expect and a recent Matrix Live interviews Amber (hawkowl) on the subject.

Aside from that we are working furiously towards federation R0 and have a bunch of MSCs to get us ever closer. You can track our progress here.

andrewsh notes that 0.34.0 is also available in the Debian repos:

Synapse 0.34.0~rc2 in Debian since Tuesday, 0.34.0 uploaded today; both use Python 3 only

🔗Dendrite

Brendan reports:

Dendrite's internal audit is progressing very well and is getting very close to its end.
What's left to do for me is check the implementation status of a few Matrix features, and translate those into tagged GitHub issues so that everyone can have as clear of a view as possible on what's left to work on.
I'm on holiday all of next week, but hopefully will have some good news about that the following week.

🔗That really is it for now

Did you get to the end? What was your favourite section? Come tell us in #twim:matrix.org! Do you have your own update you'd like to add? Same place, come chat in #twim:matrix.org.

Next week is various things. It's Christmas, which means there will be more hacking and coding happening than usual I expect! It's also 35c3, which I will be attending, and might affect scheduling next week. Stay tuned in #twim:matrix.org for news, and come join us in #matrix-35c3:matrix.org if you'll be there and want to meet up!

Merry Christmas!

Synapse 0.34.0 released!

2018-12-20 — Releases — Neil Johnson

Folks this is a big day for us at Matrix Towers, because today we release 0.34.0.

The big news for 0.34.0 is that we now recommend Python 3 for production use and have been running matrix.org under Python 3 for the past month.

Performance improvements have been marked, in some contexts we have seen 50% reductions in RAM and CPU usage. Here are some illustrative graphs to get you going but look out for a dedicated post delving into much more detail on the port. You can also see a Matrix Live interview with the project lead Amber (hawkowl) here.

Matrix.org federation reader workers, the big drops signify roll over to python 3

Synapse master on matrix.org, again the drop in RAM signifies the roll over to python 3

Many thanks to Amber for leading the effort, Rich and Erik for providing support as well as Notafile and Krombel from the community for pushing this effort right from the early days of the project.

If that wasn't enough, 0.34.0 also all the usual bug fixes and perf improvements. In particular the media repository now no longer fails to decode UTF-8 filenames when downloading remote media and auto joining rooms now work on servers with consent requirements enabled.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

In particular, if you want to run Synapse 0.34.0 on Python 3 take a look at the upgrade notes.

🔗Synapse 0.34.0 changelog

Synapse 0.34.0 is the first release to fully support Python 3. Synapse will now run on Python versions 3.5 or 3.6 (as well as 2.7). Support for Python 3.7 remains experimental.

We recommend upgrading to Python 3, but make sure to read the upgrade notes when doing so.

🔗Features

Add 'sandbox' to CSP for media reprository (#4284)
Make the new landing page prettier. (#4294)
Fix deleting E2E room keys when using old SQLite versions. (#4295)
Add a welcome page for the client API port. Credit to @krombel! (#4289)
Remove Matrix console from the default distribution (#4290)
Add option to track MAU stats (but not limit people) (#3830)
Add an option to enable recording IPs for appservice users (#3831)
Rename login type m.login.cas to m.login.sso (#4220)
Add an option to disable search for homeservers that may not be interested in it. (#4230)

🔗Bugfixes

Pushrules can now again be made with non-ASCII rule IDs. (#4165)
The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. (#4176)
URL previews now correctly decode non-UTF-8 text if the header contains a <meta http-equiv="Content-Type" header. (#4183)
Fix an issue where public consent URLs had two slashes. (#4192)
Fallback auth now accepts the session parameter on Python 3. (#4197)
Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)
fix start up failure when mau_limit_reserved_threepids set and db is postgres (#4211)
Fix auto join failures for servers that require user consent (#4223)
Fix exception caused by non-ascii event IDs (#4241)
Pushers can now be unsubscribed from on Python 3. (#4250)
Fix UnicodeDecodeError when postgres is configured to give non-English errors (#4253)

🔗Internal Changes

Debian packages utilising a virtualenv with bundled dependencies can now be built. (#4212)
Disable pager when running git-show in CI (#4291)
A coveragerc file has been added. (#4180)
Add a GitHub pull request template and add multiple issue templates (#4182)
Update README to reflect the fact that #1491 is fixed (#4188)
Run the AS senders as background processes to fix warnings (#4189)
Add some diagnostics to the tests to detect logcontext problems (#4190)
Add missing jpeg package prerequisite for OpenBSD in README. (#4193)
Add a note saying you need to manually reclaim disk space after using the Purge History API (#4200)
More logcontext checking in unittests (#4205)
Ignore __pycache__ directories in the database schema folder (#4214)
Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers (#4224)
Added automated coverage reporting to CI. (#4225)
Garbage-collect after each unit test to fix logcontext leaks (#4227)
add more detail to logging regarding "More than one row matched" error (#4234)
Drop sent_transactions table (#4244)
Add a basic .editorconfig (#4257)
Update README.rst and UPGRADE.rst for Python 3. (#4260)
Remove obsolete verbose and log_file settings from homeserver.yaml for Docker image. (#4261)

This Week in Matrix 2018-12-14

2018-12-14 — This Week in Matrix — Ben Parsons

🔗Fractal Hackfest

Developers and creators of Fractal, the GNOME Matrix client, have been holding a Hackfest this week in Seville. Matthew and I caught up with them on video for Matrix Live this week, and discussed the product improvements they've been making and their plans for the next release (due next week!)

🔗Matrix Spec

MSC1466 merged. MSC1466 from Erik describes "Soft Remote Logout", which avoids loss of encryption keys in circumstances of remote logout.
MSC1730 entered FCP (final comment period, the last step before being merged.) read the proposal

🔗This Week in Ruma

We don't often get to feature news from Ruma, but this week there is an updated This Week in Ruma.

Ruma is not dead, however, and small improvements have continued over the last year. The Matrix spec has advanced quite a bit and many of the blocking issues for Ruma have been resolved. Rust's maturity is another story. async/await is still under development and this is the most significant blocking issue to progress on Ruma.

🔗maubot new tooling

tulir has been working on tooling for maubot:

maubot got a command-line tool for building plugins and managing maubot instances. I also added some server-side stuff for easy registration of accounts in the management interface using synapse shared secrets, but the UI for that isn't ready. Also, I'm planning on adding some kind of small Matrix client in the management interface for manually managing the added bot clients. That might lead to a separate library that could be used in other projects or embedded in websites.

🔗mxisd v1.2.1 released, updating recommended

Max with a new release of mxisd:

mxisd hits v1.2.1 with a new maintenance release, fixing bugs and regressions from v1.2.0. Updating to v1.2.1 is strongly encouraged as v1.3.x will contain breaking changes and will not be a straight-forward update like v1.x has been until now.

🔗Clients

🔗Lazy-loading lands in QMatrixClient

kitsune has reported that lazy-loading, a Matrix feature that entails only loading room member details as needed, is now in libQMatrixClient master.

reference implementation in Quaternion will follow up

Black Hat has been testing the feature in Spectral, which uses the library, and says there is a 30%-50% reduction in RAM usage at startup.

🔗Riot Web progress on `/experimental` and v0.17.8 released

Bruno has been working on /experimental, and I recommend taking a look at the progress there! The next version of Riot Web is closer than ever.

Improved read markers now available on /experimental, needs further tweaking though. Brought back community UX on redesign, other small improvements.

v0.17.8 was released with several bugfixes and improvements.

🔗Riot iOS

Release made with these notes:

This new version supports the consent of matrix servers terms of service (including GDPR) in the registration flow. It also contains fixes for the "Empty room" bug, the registration issue on iOS 10, etc.

Get more information:

🔗Riot Android

Release made:

This new version supports the consent of matrix servers terms of service (including GDPR) in the registration flow. Many bugfixes SDK contains KeyBackup

Get more information:

🔗Servers

🔗Synapse

A lot of focus on getting debian packages ready for python 3 - this is a blocker for our official python 3 release 0.34.0. Aside from that, finalising some outstanding state resolution behaviour (https://github.com/matrix-org/matrix-doc/pull/1693) and dusting off event ids as hashes (https://github.com/matrix-org/matrix-doc/issues/1127.)

🔗Dendrite

Brendan:

My internal audit of Dendrite is continuing, drawing a more and more precise picture of what's left to fix and implement. I aim to have it over by the end of the year, or the very early days of 2019 at the latest. Folks can track its progress through https://cloud.abolivier.bzh/index.php/s/qXi2KFjCQk2c6eG

🔗modular.im now has Extra-Large instances available

Due to demand, modular.im Hosted Homeservers now has Extra-Large instances available. If you need to service 1,000+ users on a Matrix homeserver, this is the product for you!

🔗linuxgaming.life homeserver is the number one Matrix homeserver focused on Linux gaming

swedneck has continues his work on linuxgaming.life:

I've added some bots, health monitoring, and dimension integration manager to linuxgaming.life, and made the website dark.

The homeserver is open for new registrations.

🔗That's all I know!

If you'd like see your Matrix-related project featured in this blog post, come chat to us in #twim:matrix.org, and we'll see you next week!

This Week in Matrix 2018-12-07

2018-12-07 — This Week in Matrix — Ben Parsons

🔗Matrix Live S03E06

In which Matthew & Amandine discuss The Matrix.org Foundation, go-live for the French Government deployment for Matrix, and some exciting random diversions into post-1.0 Matrix features which should land once once 1.0 is out the door!

🔗Purism sponsoring Fractal development

We are proudly sponsoring the work of Julian Sparber on Fractal, the @matrixdotorg client for GNOME. Read his latest update - https://t.co/q4ubjzETxC #LibreDesktop #DemandFreedom #gnome

— Purism (@Puri_sm) December 5, 2018

Purism announced they are sponsoring Fractal development, starting with Julian Sparber working last month to improve the message view. You can read about his progress here. Highlights:

Before, there was a jarring cut when new messages were loaded, but now you can just scroll upward and older messages are loaded continuously.
The part I'm most excited about is the new "new message divider". When the user opens a room they can directly start reading the conversation from the last seen message and they don't need to search for the new message divider. Not only has the UX gotten a lot better, but also the underlying code is much cleaner now.
I also spent some time on making message rendering faster. I replaced the RegExp with more efficient code, this made the rendering much faster (from ~10ms to ~ 1ms) for every single message.
In summary, all of these things improve Fractal's UX a lot and make it feel more like a modern messaging app.

🔗Synapse 0.34.0rc1

0.34.0rc1 is out - please test it! 0.34.0 will be the first to officially support python 3. We're still working on the debian packaging but 0.34.0 proper should be out next week.

🔗Dendrite

Brendan has been getting on with Dendrite development:

Started auditing Dendrite's codebase to identify what is left to implement, along with what hasn't been implemented correctly. Still a lot of work left to do on this, but it's looking promising so far.

🔗Modular

Modular is a Hosted Homeservers product, check out modular.im. This week:

Matrix -> Matrix account migration tool available at https://www.modular.im/tools/matrix-migration. It's still beta, so please test it and let us know if you experience issues.

🔗New Homeserver: linuxgaming.life

swedneck has set up a new public homeserver at linuxgaming.life! There is also a Riot installation at https://riot.linuxgaming.life, served via IPFS.

Currently working on the website (based on t2bot.io, thank you travis for making that available on github!) and making things nicer in general.
I will be adding more bots shortly. Please let me know of any issues with the HS or anything surrounding it.

🔗Informo

Informo is a project to create a specification and implementation for distributing information and news. This week they have been working on merging changes to the specification:

Work on the Informo specifications has slowed down a bit in the last couple of weeks, though since the last update we did manage to get some relatively big SCSs merged into the specs, including SCS #9 (rendered here) which specifies how information sources must publish their information through the federation, and SCS #11 (rendered here) which describes how sources must register themselves in order to be picked up by clients, and handle localisation. ?
There's still a couple of big items to take care of before we can cut a 1.0 release of the Informo specs but this is definitely a huge step towards this goal.

🔗Minecraft Bridge

Dandellion:

I started working on a Minecraft bridge pretty heavily based on Travis's old project, but using Minecraft-protocol instead of mineflayer.
You need a bot Minecraft account that can join and idle in the server, it then uses /tellraw to post messages.

🔗libQMatrixClient 0.4.1 released

libQMatrixClient, which powers Spectral and Quaternion was released by kitsune, Lazy Loading coming soon:

libQMatrixClient 0.4.1 has been released today, with small fixes in the stable branch. Meanwhile, active work is ongoing on lazy-loading support in the library, with ETA for the feature landing in master being in about a week or so.

🔗Riot Web

Redesign progress: all resizeable sections are now persisted, (slow) progress on improving read markers
Communities and community invitations should be more resilient to errors (on /develop only so far)

🔗Riot iOS

Reskin still ongoing
A new release is coming
Less activity because of POSS

🔗Riot Android

A new developer joined us to help maintaining Riot: Valere Fedronic
Keys backup PR in review
Privacy Terms acceptance in login flow in review
FCM issue investigation/troubleshooting by Valere
François Off (sick :()
New theme is coming soon

🔗Koma

druig has been working on Koma, a JavaFX Matrix client:

This week in koma: implementation of json library switched from runtime reflection to compile-time code generation
The matrix api is implemented manually in the project, retrofit and moshi are used to interact with the rest api.

🔗SimpleMatrix

As we mentioned in TWIM last week, MTRNord has been working on designs for SimpleMatrix, a Matrix client for Android in development. This week he has made a video showcasing the new design.

🔗Redecentralize meetup in London

Last night Half-Shot & Neil & Brendan went to see the Redecentralize folk at their meetup:

Beer! And pizza! And also Redecentralize
We chatted to a nice bunch of folks both demoing their decentralized projects and talked to the likes of scuttlebutt, BBC, IPFS and more. Was a interesting experience having around 8 minutes to quickly explain to newcomers what Matrix is and why they should use it in a speed dating format. We chatted (and demoed) bridges, new-riot and generally how it all fits together. There were a lot of very interesting people with different profiles and backgrounds, and diversified questions which lead to a lot of interesting discussions.

Note for the confused: the format of the meetup was comparable to speed dating. There is no suggestion that Matrix should be used at regular speed dating. If you do find such a use for Matrix, come tell us in #twim:matrix.org.

🔗matrix-wug, X-SAMPA to IPA bot

Last month Dandellion introduced a bot designed to convert between formats for pronunciation notation, this week he released the source code: https://github.com/dali99/matrix-wug

🔗The end of the post, and nearly the end of 2018

This week it has been winter-in-Europe kind of weather, but that's ok, it happens every year. TWIM on the other hand happens every week, so if you have something to share, and would like to share what you've been working on, come chat to us in #twim:matrix.org.

This Week in Matrix 2018-11-30

2018-11-30 — This Week in Matrix — Ben Parsons

🔗Porting to Python 3 - Matrix Live S3E5

Longer than usual episode of Matrix Live this week. Amber talks to Neil about the works she's been doing over the last few months to port Synapse from Python 2 to Python 3. Recommended for anyone who's been following along the progress of Synapse, or who wants a good intro on the benefits of Python 3 over Python 2.

🔗ZeroPhone

This week we learned about ZeroPhone, a fairly mature project aiming to build a working cellphone on top of a Raspberry Pi Zero.

an open-source smartphone that can be assembled for 50$ in parts. It is Linux-powered, with UI software written in Python. Currently, ZeroPhone is based on Raspberry Pi Zero, SIM800L GSM modem and 1.3" OLED screen and button interface.

&Adam was the first to bring it to our attention:

I stumbled across this blog post which mentions a matrix app in a project to create an open, hackable phone out of a raspberry pi. It's old news at this point I guess, but I don't think I've seen it mentioned here, so I thought it might be a neat thing worth featuring: https://zerophone.github.io/newsletter/ZeroPhone-Weekly-No.-16/

The creator of the Matrix app, derivmug also arrived to tell us about his work:

Hey everyone, I have been suggested to post about the Zerophone project here. I have written a basic matrix client for it. It's basically a simple phone based on a raspberry pi zero, focused on privacy, security and hackability.

The client app is built using the matrix-python-sdk, I'm also hoping to chat to derivmug as part of a future Matrix Live. Take a look at the code here.

Just to prove it's running on real hardware!

🔗Spec / Foundation

Work continues on MSC1730 - Mechanism for redirecting to an alternative server during login https://github.com/matrix-org/matrix-doc/pull/1730
Foundation: Working with the Foundation lawyers on the CIC37 to register the Foundation as a Community Interest Company proper

🔗The Federation (the-federation.info)

Jason Robinson is passionate about Federated technology, and has been working on his site at https://the-federation.info/ for some time. He recently added Matrix to the list of protocols he scrapes and lists:

Matrix (or more specifically, Synapse) servers are now visible on the The Federation info website at https://the-federation.info/matrix%7Csynapse. The website collects lists of servers for various federation protocols (like Matrix, ActivityPub, Diaspora, etc). If a server outputs usage information, it also shows some historical information for servers and projects. Currently the information available from Synapse on the site is just version, availability of registrations and (obviously) domain name. If you want to register your server, check https://the-federation.info/info for more details. Tips also welcome on how to register other Matrix server projects and how to get more data out of servers.

🔗maubot

tulir has continued work on maubot:

I made a rss plugin for maubot and added a fancy log viewer in the maubot management UI.

mautrix-telegram 0.4.0 was released, though it has no changes since the release candidate. I also copied mautrix-telegram's HTML parser into mautrix-python. Hopefully it's now generic enough so that when mautrix-telegram switches to mautrix-python, it can drop most of the built-in HTML parser.

🔗matrix-appservice-discord

Bridge-of-the-week this week is matrix-appservice-discord. Anyone in the dev room (#discord:half-shot.uk) will be very aware of this as Half-Shot and Sorunome have spent the day testing @room notifications.

Sorunome:

matrix-appservice-discord received a lot of work these past days. Not only the usual small bugfixes, but also some rather big changes: The parsers for both Discord->Matrix sending and Matrix->Discord sending were re-written completely to properly handle things (and thus, at the same time, crunch a bunch of edge-case bugs!). In addition, the long-awaited highlighting issues with @ everyone and @ here (on discord) and @ room (on matrix) were addressed. Some of these things are still in PR stage, but expected to be merged soon!

Half-Shot also provided a great screenshot of everything working well:

is just showing off having IRC+Matrix+Discord+XMPP plugged together. XMPP being the one purple is bridging

🔗matrix-docker-ansible-deploy

Slavi reports that:

matrix-docker-ansible-deploy now supports configuring the matrix-synapse-ldap3 LDAP auth password provider. Thanks to @tvo6 for contributing this!

🔗Riot iOS

User Agreement (GDPR) in now part of the registration flow
Reskin has started
New fixes for unexpected “Empty room”
We have PR templates

🔗Riot Android

Realm store for e2e keys has landed
Riot reboot: support timelines merge in DB to avoid to trash data as less as possible when navigating through permalinks
New settings to choose ringtone for incoming call
We have PR templates also

🔗SimpleMatrix

MTRNord, is working on the design for SimpleMatrix:

As I currently have less time for actual coding I am planning out the SimpleMatrix Design. You can find the Images of the Design (and a PDF file) at https://gitlab.com/Nordgedanken/simplematrixredesignimages Feel free to also join #SimpleMatrix:matrix.ffslfl.net to give feedback.

🔗Synapse

This week our focus has been to work through some final bugs blocking 0.34.0 which will be our first official python 3 release. Expect a RC rsn. For a sneak peak see this week's Matrix Live video at the top!

🔗Auf Wiedersehen…

That's it for this week. If you have anything to share with us, and you'd like to be included on the blog, please come talk in #twim:matrix.org!

In one month from now, many Matrix-acolytes, including myself, will be at 35c3. If you'll be there, come chat to us in #matrix-35c3:matrix.org!

This Week in Matrix 2018-11-26

2018-11-26 — This Week in Matrix — Ben Parsons

A little delay to the release of this week's TWIM, expect us to resume normal schedule on Friday.

🔗Spec & Foundation

Two workshops happened with the Foundation lawyers to work on MSC1318 - basically fleshing out a proposal for the details of the foundation governance in terms of how the Guardians (i.e. directors of the foundation board) should interact with the Core Spec Team to act as a back-stop - and working out how to port the MSC (once finished) into formal Articles of Association & adjunct documents. We'll get the workshop notes written up as a rewrite of MSC1318 very shortly.
New MSCs from vdH on:
- MSC1721 Rename m.login.cas to m.login.sso
- MSC1730 Mechanism for redirecting to an alternative server during login
…and an alternate to MSC1730: MSC1731 Mechanism for redirecting to an alternative server during SSO login.
These three SSO-related MSCs are derived from work being done for a large-scale Matrix deployment for a New Vector customer which we should be able to talk about in future. Improving SSO and other auth/identity work in Matrix is vital, and needless to say we're trying to ensure all the resulting work ends up in the spec to the benefit of the whole ecosystem :)

🔗matrix-media-repo

TravisR reports that:

matrix-media-repo has received a lot of updates in the last few weeks. Although mostly bug fixes and improvements, there's added support for streaming downloads from remote servers. This is mostly useful when someone posts a 100mb video in a room: previously the media repo would download the entire thing then send it over to you. Now it'll send you the video while it also caches it.

🔗matrix-docker-ansible-deploy

Big update from Slavi this week:

matrix-docker-ansible-deploy has received some updates in the past few weeks - fixing minor bugs, improving documentation, adding support for additional Synapse/Riot features, etc.

Most notably (in oldest to most-recent order):

Cadair heard you like playbooks, so he made it possible to run the playbook from another playbook. Thanks Cadair!

some configuration renaming to make things more consistent. Most people would likely be unaffected.

support has been added for Postgres 11. Upgrading is possible and easy (as a related success story, Aaron Raimist shared that he recently upgraded his ~40GB database)

more flexible and more secure-by-default TLS configuration. Thanks Aaron Raimist!

the ability to customize more parts of riot-web. Thanks Aaron Raimist!

better control over Synapse logging. Thanks Aaron Raimist!

the ability to set up a public server and the ability to make use of Synapse's \\"auto-join\\" feature. Until now, the playbook was only suited for setting up a private server, but with this, setting up public servers is now a possibility. Thanks @anadahz

I'm quite happy that the playbook is getting more contributions from others now and is improving in such diverse ways!

Additionally, the #matrix-docker-ansible-deploy:devture.com support room in Matrix is now bridged to the #matrix-docker-ansible-deploy IRC Freenode channel, making it easier for people to get support when they don't have a working homeserver.

🔗libQMatrixClient

kitsune has been working on libQMatrixClient:

the work on lazy-loading in libQMatrixClient has advanced to the point when client authors may start trying to integrate it in their applications (welcome to #qmatrixclient:matrix.org for details - it's a bit rough around the edges yet).

🔗Spectral

This week Black Hat has been working to give Spectral a more Material design oriented UI.

🔗SMSMatrix

swedneck reports that:

SMSMatrix now has a room (#smsmatrix:matrix.org). SMSMatrix is a simple bridge between Matrix and SMS, developed by tijder. You install an android app and register a bot account which will invite you to direct chats under the name of anyone who sends you an SMS, which will then act as a normal SMS conversation.

I had a very quick play with this tool and it works great - a good way to quickly bridge SMS to Matrix.

🔗Riot

News from the Riot teams:

Android rewrite is continuing in Kotlin - lots of infrastructure work over the last few weeks
E2E key backups landed in Riot/iOS develop (but no UI yet, as we finalise the implementation first on Riot/Web)
E2E key backups also landed in Riot/Web develop (but with placeholder UI)
E2E key backups in dev on Riot/Android too (using Realm for storage!)
Lots of work from uhoreg on E2E cross-signing
Work on the Riot/Web redesign paused for a week due to other commitments but as of Nov 26th we're back on the case again, making up for lost time.

🔗Synapse

Most of the focus from the Synapse team this week has been on the new version using Python 3:

Py3 is finished, other than Debian packages - ongoing debate on whether they should use a virtualenv or depend on debian dependencies, and who should maintain them in a Py3 world…
Lots of running around on final Py3 bugs

🔗matrix-appservice-purple

Half-Shot has continued work on matrix-appservice-purple.

🔗Modular

Modular.im provides Hosted Homeservers for Matrix.

Modular now has full support for Custom DNS (letting you point your own SRV records at it)
Lots of work on a migration tool to allow folks to easily shift Matrix accounts between servers, as well as (in future) migrations from HipChat, Slack & others.

🔗That's it!

Apologies for the delayed release this week - please expect an episode of Matrix Live in the week, and back to a normal schedule for TWIM as of Friday. As always, come talk to us in #thisweekinmatrix:matrix.org and tell us what you've been doing.

Synapse 0.33.9 is here!

2018-11-20 — Releases — Neil Johnson

Well here you are then, the 9th episode in the Synapse 0.33.x series.

Features wise, 0.33.9 contains a change to the way that GDPR consent works under the hood. It is now plumbed in to the login flow (rather than following immediately afterwards) such that it does not inadvertently break on-boarding. This is part of a broader set of changes that span Synapse and Riot to improve initial first impressions of using matrix.

Separately we now have support for room version upgrades which is pre-requisite for rolling out the new state resolution algorithm, come and join us in #teststateresv2:jki.re if you would like to help us test.

Finally we've spent a bunch of time further improving perf especially in and around reducing device ids federation traffic.

I know I say it every time, but full python 3 support is really really close now, matrix.org is now running entirely on py3 and seeing some amazing perf improvements - the remaining blocker is getting py3 deb packages ready and then we'll ship an official python 3 release. There will also be a blog post to explain what we've been up to and what to expect perf wise.