Technology and Art

- Deep Neural Networks as Gaussian Processes. (Also see this)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- A ConvNet for the 2020s
- InstructPix2Pix: Learning to Follow Image Editing Instructions

This post is probably a spiritual successor to Resilient Knowledge Bases.

This article continues from where Every Software Engineer is an Accountant left off. I have had feedback that I need to make my posts on these a little more explainable; I will attempt to do that here.

This is a weird mix of advice I’d give the less-experienced me, as well as reflections of my personal value system. This verbal diarrhoea came out all at once in a single sitting of 45 minutes. I apologise for some of the strong language in here, but I thought I’d share it without much censoring.

This article continues from where Every Software Engineer is an Economist left off, and delves slightly deeper into some of the topics already introduced there, as well as several new ones. In the spirit of continuing the theme of “Every Software Engineer is an X”, we’ve chosen accounting as the next profession.

**Background**: This post took me a while to write: much of this is motivated by problems that I’ve noticed teams facing day-to-day at work. To be clear, this post does not offer a solution; only some thoughts, and maybe a path forward in aligning developers’ and architects’ thinking more closely with the frameworks used by people controlling the purse-strings of software development projects.

We continue looking at the **Transformer** architecture from where we left from Part 1. When we’d stopped, we’d set up the Encoder stack, but had stopped short of adding positional encoding, and starting work on the Decoder stack. In this post, we will focus on setting up the training cycle.

**TL;DR:** I went to using Vim once in a year to using it everyday by accident when I got into a flow mindset after the effort of understanding a Machine Learning paper. It feels like a miracle.

**Note:** This is a post from July 13, 2011, rescued from my old blog. This is only for archival purposes, and is reproduced verbatim, but I make no claims about its rigour, though it does still seem plausible.

**Note:** This is a post from July 13, 2011, rescued from my old blog. This is only for archival purposes, and is reproduced verbatim, but is hopelessly outdated.

This short post lists the Neovim (Vim) shortcuts I am getting used to. I’ve recently switched to trying the Vim mode for my IDE needs, and having used Vim previously only for very simple tasks, am having a blast practising the basic Vim shortcuts. Ultimately, I will probably move to doing more IDE-related work in native Vim too.

This is part of a series of posts breaking down the paper Plenoxels: Radiance Fields without Neural Networks, and providing (hopefully) well-annotated source code to aid in understanding.

This is part of a series of posts breaking down the paper Plenoxels: Radiance Fields without Neural Networks, and providing (hopefully) well-annotated source code to aid in understanding.

This is part of a series of posts breaking down the paper Plenoxels: Radiance Fields without Neural Networks, and providing (hopefully) well-annotated source code to aid in understanding.

It may seem strange that I’m jumping from implementing a simple neural network into **Transformers**. I will return to building up the foundations of neural networks soon enough: for the moment, let’s build a **Transformer** using PyTorch.

Programming guides are probably the first posts to become obsolete, as APIs are updated. Regardless, we will look at building simple neural networks in **PyTorch**. We won’t be starting from models with a million parameters, however. We will proceed from the basics, starting with a single neuron, talk a little about the tensor notation and how that relates to our usual mathematical notation of representing everything with column vectors, and scale up from there.

This is a quick note on proving the **Triangle Inequality** criterion of the following claim:

This post lists solutions to the exercises in the **Linear Operators section 2.6** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This post lists solutions to the exercises in the **Compactness and Finite Dimension section 2.5** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This post lists solutions to the exercises in the **Finite Dimensional Normed Spaces and Subspaces section 2.4** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This post lists solutions to the exercises in the **Further Properties of Normed Spaces section 2.3** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

We review the most interesting failure scenario for the **Two Phase Commit (2PC) protocol**. There are excellent explanations of 2PC out there, and I won’t bother too much with the basic explanation. The focus of this post is a walkthrough of the **indistinguishable state scenario**, where neither a global commit, nor a global abort command can be issued.

This post lists solutions to the exercises in the **Normed Space, Banach Space section 2.2** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

We start this tirade with a memorable quote from **Alice Through the Looking-Glass**:

This post lists solutions to the exercises in the **Vector Space section 2.1** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This post lists solutions to the exercises in the **Completion of Metric Spaces section 1.6** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This post lists solutions to the exercises in the **Completeness Proofs section 1.5** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This post lists solutions to the exercises in the **Convergence, Cauchy Sequences, and Completeness section 1.4** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This post lists assorted proofs from **Analysis**, without any particular theme.

This post lists solutions to many of the exercises in the **Open Set, Closed Set, Neighbourhood section 1.3** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

This article represents a (very short) collection of my **ongoing notes on proof tactics** I’ve found useful when I’ve been stuck trying to solve proof exercises. I aim to continue documenting these in as much detail as possible. These are mostly aids while building intuition about how to prove something, and gradually should become part of one’s mental lexicon.

This post lists solutions to many of the exercises in the **Distance Metrics section 1.2** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is a work in progress, and proofs may be refined over time.

Continuing my self-study of **Functional Analysis**, this post describes proofs for the following important inequalities in the subject:

- Young’s Inequality
- Hölder’s Inequality
- Minkowski’s Inequality

This post lists solutions to many of the exercises in the **Distance Metrics section 1.1** of *Erwin Kreyszig’s* **Introductory Functional Analysis with Applications**. This is definitely a work in progress, and proofs may be refined or added over time.

In this article, we will build up our mathematical understanding of **Gaussian Processes**. We will understand the conditioning operation a bit more, since that is the backbone of inferring the **posterior distribution**. We will also look at how the **covariance matrix** evolves as training points are added.

In this article, we will build up our intuition of Gaussian Processes, and try to understand how it models uncertainty about data it has not encountered yet, while still being useful for regression. We will also see why the **Covariance Matrix** (and consequently, the **Kernel**) is a fundamental building block of our assumptions around the data we are trying to model.

Continuing from the roadmap set out in Road to Gaussian Processes, we begin with the geometry of the central object which underlies this Machine Learning Technique, the **Multivariate Gaussian Distribution**. We will study its form to build up some geometric intuition around its interpretation.

This article covers some common statistical quantities/metrics which can be derived from **Linear Algebra** and corresponding intuitions from **Geometry**, **without recourse to Probability or Calculus**. Of course, those subjects add more rigour and insight into these concepts, but our aim is to provide a form of intuitive shorthand for the reader.

This article builds upon the previous material on **kernels** and **Support Vector Machines** to introduce some simple examples of **Reproducing Kernels**, including a simplified version of the frequently-used **Radial Basis Function kernel**. Beyond that, we finally look at the actual application of kernels and the so-called **Kernel Trick** to avoid expensive computation of projections of data points into higher-dimensional space, when working with **Support Vector Machines**.

This article uses the previous mathematical groundwork to discuss the construction of **Reproducing Kernel Hilbert Spaces**. We’ll make several assumptions that have been proved and discussed in those articles. There are multiple ways of discussing Kernel Functions, like the **Moore–Aronszajn Theorem** and **Mercer’s Theorem**. We may discuss some of those approaches in the future, but here we will focus on the constructive approach here to characterise **Kernel Functions**.

This article expands the groundwork laid in Kernel Functions: Functional Analysis and Linear Algebra Preliminaries to discuss some more properties and proofs for some of the properties of functions that we will use in future discussions on **Kernel Methods** in Machine Learning, including (but not restricted to) the construction of **Reproducing Kernel Hilbert Spaces**.

These are personal study notes, brief or expanded, complete or incomplete. Some concepts here will be alluded to in full-fledged Machine Learning posts.

This article lays the groundwork for an important construction called **Reproducing Kernel Hilbert Spaces**, which allows a certain class of functions (called **Kernel Functions**) to be a valid representation of an **inner product** in (potentially) higher-dimensional space. This construction will allow us to perform the necessary higher-dimensional computations, without projecting every point in our data set into higher dimensions, explicitly, in the case of **Non-Linear Support Vector Machines**, which will be discussed in the upcoming article.

ld
Continuing on my journey through **Real Analysis**, we will focus here on common **proof patterns** which apply to **irrational square roots**.
These patterns apply to the following sort of proof exercises:

This article discusses a set of two useful (and closely related) factorisations for **positive-definite matrices**: the **Cholesky** and the **\(LDL^T\)** factorisations. Both of them find various uses: the Cholesky factorisation particularly is used when **solving large systems of linear equations**.

We discuss an important factorisation of a matrix, which allows us to convert a linearly independent but non-orthogonal basis to a **linearly independent orthonormal basis**. This uses a procedure which iteratively extracts vectors which are orthonormal to the previously-extracted vectors, to ultimately define the orthonormal basis. This is called the **Gram-Schmidt Orthogonalisation**, and we will also show a proof for this.

Since I’m currently self-studying **Real Analysis**, I’ll be listing down proofs I either initially had trouble understanding, or enjoyed proving, here. These are very mathematical posts, and are for personal documentation, mostly.

We have looked at how **Lagrangian Multipliers** and how they help build constraints as part of the function that we wish to optimise. Their relevance in **Support Vector Machines** is how the constraints about the classifier margin (i.e., the supporting hyperplanes) is incorporated in the search for the **optimal hyperplane**.

This article concludes the (very abbreviated) theoretical background required to understand **Quadratic Optimisation**. Here, we extend the **Lagrangian Multipliers** approach, which in its current form, admits only equality constraints. We will extend it to allow constraints which can be expressed as inequalities.

This article continues the original discussion on **Quadratic Optimisation**, where we considered **Principal Components Analysis** as a motivation. Originally, this article was going to begin delving into the **Lagrangian Dual** and the **Karush-Kuhn-Tucker Theorem**, but the requisite mathematical machinery to understand some of the concepts necessitated breaking the preliminary setup into its own separate article (which you’re now reading).

We consider the more frequently utilised viewpoints of **matrix multiplication**, and relate it to one or more applications where using a certain viewpoint is more useful. These are the viewpoints we will consider.

We discussed the **Implicit Function Theorem** at the end of the article on Lagrange Multipliers, with some hand-waving to justify the linear behaviour on manifolds in arbitrary \(\mathbb{R}^N\).

We pick up from where we left off in Quadratic Optimisation using Principal Component Analysis as Motivation: Part One. We treated **Principal Component Analysis** as an optimisation, and took a detour to build our geometric intuition behind **Lagrange Multipliers**, wading through its proof to some level.

In this article, we finally put all our understanding of **Vector Calculus** to use by showing why and how **Lagrange Multipliers** work. We will be focusing on several important ideas, but the most important one is around the **linearisation of spaces at a local level**, which might not be smooth globally. The **Implicit Function Theorem** will provide a strong statement around the conditions necessary to satisfy this.

In this article, we take a detour to understand the mathematical intuition behind **Constrained Optimisation**, and more specifically the method of **Lagrangian multipliers**. We have been discussing **Linear Algebra**, specifically matrices, for quite a bit now. **Optimisation theory**, and **Quadratic Optimisation** as well, relies heavily on **Vector Calculus** for many of its results and proofs.

This series of articles presents the intuition behind the **Quadratic Form of a Matrix**, as well as its optimisation counterpart, **Quadratic Optimisation**, motivated by the example of **Principal Components Analysis**. PCA is presented here, not in its own right, but as an application of these two concepts. PCA proper will be presented in another article where we will discuss **eigendecomposition**, **eigenvalues**, and **eigenvectors**.

This article aims to start the road towards a theoretical intuition behind **Gaussian Processes**, another Machine Learning technique based on **Bayes’ Rule**. However, there is a raft of material that I needed to understand and relearn before fully appreciating some of the underpinnings of this technique.

We will derive the intuition behind **Support Vector Machines** from first principles. This will involve deriving some basic vector algebra proofs, including exploring some intuitions behind hyperplanes. Then we’ll continue adding to our understanding the concepts behind quadratic optimisation.

The **dot product of two vectors** is geometrically simple: the product of the magnitudes of these vectors multiplied by the cosine of the angle between them. What is not immediately obvious is the algebraic interpretation of the dot product.

Let’s look at **Linear Regression**. The “linear” term refers to the fact that the output variable is a **linear combination** of the input variables.

I’d like to introduce some basic results about the rank of a matrix. Simply put, the rank of a matrix is the number of independent vectors in a matrix. Note that I didn’t say whether these are column vectors or row vectors; that’s because of the following section which will narrow down the specific cases (we will also prove that these numbers are equal for any matrix).

Some of these points about matrices are worth noting down, as aids to intuition. I might expand on some of these points into their own posts.

We will discuss the Column-into-Rows computation technique for matrix outer products. This will lead us to one of the important factorisations (the **LU Decomposition**) that is used computationally when solving systems of equations, or computing matrix inverses.

This is the easiest way I’ve been able to explain to myself around the orthogonality of matrix spaces. The argument will essentially be based on the geometry of planes which extends naturally to hyperplanes.

We will discuss the value-wise computation technique for matrix outer products. This will lead us to a simple sketch of the proof of reversal of order for transposed outer products.

Matrix multiplication (outer product) is a fundamental operation in almost any Machine Learning proof, statement, or computation. Much insight may be gleaned by looking at different ways of looking matrix multiplication. In this post, we will look at one (and possibly the most important) interpretation: namely, the **linear combination of vectors**.

Linear Algebra deals with matrices. But that is missing the point, because the more fundamental component of a matrix is what will allow us to build our intuition on this subject. This component is the vector, and in this post, I will introduce vectors, along with common notations of expression.

I’ve always been fascinated by Machine Learning. This began in the seventh standard when I discovered a second-hand book on Neural Networks for my ZX Spectrum.

My previous Wordpress site went kaput. With it went most of the content that I’ve posted over the years. I have switched over to Jekyll, and will be adding new material, as well as reintroducing old material as needed, from a backup that I have. Stay posted!