Mute Buttons Are The Latest Discourse Markers

Every one, and their mom, is on a video call at least once a day now. There is a tiny second-order effect brewing in these video calls. It’s do with the mute buttons.

Let me describe a common-place scenario in conference calls. In a call with a fair number¹ of participants, we tend to keep our microphones muted. It’s common etiquette, and the reason to do so is pretty plain. You do not want to burden the speaker & other participants with your background noise; and this helps to keep the conversations as distraction-free as possible. So, what happens when you want to speak? You unmute the microphone. Simple! And once you have spoken, you “concede the conversation” by muting yourself back.

In short, mute buttons are functioning as discourse markers, and are our latest language innovation. This also reminds me a bit of Hyrum’s Law:

… all observable behaviors of your system will be depended on by somebody.

The Blue Flower : A Review

The Blue Flower by Penelope Fitzgerald
My rating: 5 of 5 stars

Historical fiction can work at such disparate levels; an era as a backdrop for the narrative, familiar textbook history unraveling as background score to the symphony of the lead characters’ life, the idiosyncrasies of the bygone era pictured in contrast to the era of the writer. The Blue Flower has everyone of these devices used to perfection, but it is so much more. .. Read More

S3 and HDFS

Cluster storage systems have, over the past decade, moved their gold standards from directory-oriented file-systems such as HDFS to object-stores such as AWS S3. The two storage models have been dissected & compared over & again from multiple perspectives ^{1 2}. Again, based on your use-case, you might be more interested in a certain cross-section of differences between S3 & HDFS than other differences. I am not trying here to repeat the analyses.

I wrote this short, bullet-style compilation as a quick refresher for myself on ways S3 differs from HDFS; it is focused on the APIs & interactions Hadoop-like data-processing systems (such as Hadoop, Spark, or Flink ) might have with storage systems.

Converse Conway's Law

Melvin Conway in his 1968 paper How Do Committees Invent? postulated the now-famous Conway’s Law.

Organisations which design systems are constrained to produce designs which are copies of the communication structures of these organisations.

This homomorphism between organisational communication structures and systems designed by them, has become an adage in software management. It implies a one-way effect, though. But, does it work in the other direction?

Providing Streaming Joins as a Service at Facebook

Providing Streaming Joins as a Service at Facebook. Jacques-Silva, G., Lei, R., Cheng, L., et al. (2018). Proceedings of the VLDB Endowment, 11(12), 1809-1821.

Stream-stream joins are a hard problem to solve at scale. “Providing Streaming Joins as a Service at Facebook” provides us the overview of systems within Facebook to support stream-stream joins.

The key contributions of the paper are:

1. a stream synchronization scheme based on event-time to pace the parsing of new data and reduce memory consumption,

2. a query planner which produces streaming join plans that support application updates, and

3. a stream time estimation scheme that handles the variations on the distribution of event-times observed in real-world streams and achieves high join accuracy.

Trade-offs in Stream-Stream Joins

Stream-stream joins have a 3-way trade-off of output latency, join accuracy, and memory footprint. One extreme of this trade-off is to provide best-effort (in terms of join accuracy) processing-time joins. Another extreme is to persist metadata associated with every joinable event on a replicated distributed store to ensure that all joinable events get matched. This approach provides excellent guarantees on output latency & join accuracy, with memory footprint sky-rocketing for large, time-skewed streams.

The approach of the paper is in the middle: it is best-effort with a facility to improve join accuracy by pacing the consumption of the input streams based on dynamically estimated watermarks on event-time.