I use the terraform import feature to copy configurations from test accounts, and need to do so for all dependencies of one resource being “imported”, to avoid missing configs.
Since this happens iteratively, running terraform init every run to get the providers installed was a bad idea since 1) redundant 2) less efficient.

Plugin Cache Dir

terraform init was maintaining separate copies of the plugins and were being downloaded for each configuration.

Terraform optionally allows the use of a local directory as a shared plugin cache which can be used as the central plugins directory, thus avoiding downloading the plugins every time.

To enable the plugin cache, use the plugin_cache_dir setting in the CLI configuration file .terraformrc. For example:

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

The plugin-cache directory needs to be created already since the init command will not create it if not found. Terraform will then use. the installed plugins from this directory if present already, and install in the plugin cache if the directory is empty

OR

terraform init -plugin-dir=$HOME/.terraform.d/plugin-cache

FileSystem Mirror configuration

We could also use a filesystem mirror method to explicitly install plugins using a local directory with the option to include/exclude specific providers.

provider_installation {
  filesystem_mirror {
    path    = "/usr/terraform.d/providers"
    include = ["example.com/*/*"]
  }
  direct {
    exclude = ["example.com/*/*"]
  }
}

Both worked well for the above case, with plugin cache being a straightforward approach 👍

Intrigued, I delved into the logic to investigate. Initially, everything seemed in order. For instance, a file named “blöb” was saved precisely as “blöb” — no bizarre encoding alterations were apparent.

Can you spot any difference between “blöb” and “blöb”?

This time instead of typing the word in the search bar, I copied it from the uploaded and saved filename, and and voilà, the search functioned perfectly! 🤯. This definitely meant the file name was subtly altered upon upload, hinting at a background encoding transformation.

So, as anyone would, I employed the power of encodeURIComponent to see for myself what really was different.

There it was! The different types of encoding forms that led to this. Now you might be wondering (I was wondering too!):

As if worrying about encoding wasn’t enough, now we’ve got to stress over its different types too 🤦‍♀ and wait, its 2024, and we are still grappling with Unicode character encoding problems?

The culprits in this case were these two contrasting encoding types: %C3%BC and %CC%88, leading to the search malfunction. This is a well-known issue with NFD/NFC encoding forms.
There is a detailed explanation here.

Now imagine explaining this technical jargon to a user!😃

To fix it, I aligned the encoding type of the searched filename with that of the saved filename.
On MacOS, the encoding was NFD (for ö: 111 + 776), whereas on Linux, it switched to NFC (ö: 246).

The solution?

– Post-upload, before saving the file as an entity, I applied .normalize() to the filename, ensuring it was uniformly searchable.

– Additionally, as a cleanup step — to write a migration script to normalize the names of all existing files with non-ASCII characters.

A simple string normalise proved effective in this case. ✅
Read more about String.prototype.normalize() here. There is also a specific normalization option to pass to ensure canonically equivalent normalization — for example: string.normalize("NFC")

Here’s a visual representation of the problem with the solution output:

https://drive.google.com/file/d/1Jkp8ye47iRfDoeTaMQkkskY_BTHegj_D/view

Hope it helps!

What one’d do in such a situation is look at the obvious things first: are the images loaded lazily? Is the bundle size too big? Is the main thread blocked? How is the performance panel recording looking? Is the data coming from the backend taking more time than it should?

If it’s one of the first problems, we’d fire up our favorite devtools and try to find out the root cause. If it’s the data being slow, we could see how long that is taking in the network requests duration. And, if it does end up being slow from the backend, the next step would be to dive into some logs and checking what piece of code in the request endpoint takes the time, or embedding some performance information in the response, or using other performance tools etc.

We had a similar situation at epilot where we use Elasticsearch to look for all related products of an order for example, and also all the other entities the order itself might be a relation to.

This meant, looking for direct relations of the order but also the reverse relations of that order.

Now in the Elasticsearch world, this meant:

1. Getting the list of related products of the order.
2. Searching through the documents in elasticsearch for all those related IDs.
3. Also, searching all other docs for having this one order as a relation to this (the reverse relation example, I shared above).
4. Finally returning all the found relations.

Showcasing two products as relations of an order

Entities and relationships

This worked great for the first few years and got us exactly what we needed quickly. However, as we scaled, the amount of documents indexed in ES increased too, and that meant, the step 3 above started getting expensive and slow! The logic was to look for an entity ID in all the indexed documents and find if it is a relation or not. For orders with a lot of products, the endpoint would respond in seconds.

We had to fix the logic/ update the design asap.

The first solution was to update the ES query to make it more specific about what is being looked up, and then check if it improved with that query change and how the results looked for the entities with many relations. This meant looking into a crazy amount of Cloudwatch logs and looking at the total time taken and if it was any better.

We made a change to search query to be an IDs specific query which would look through only the IDs in the indexed docs, and not through all the fields in those ES documents.

GET /_search
{
  "query": {
    "ids" : {
      "values" : ["1", "4", "100"]
    }
  }
}

The expectation was this would completely solve the problem and be super quick. This wasn’t the case unfortunately.

What we were missing was to see which part of the code was really adding to the time taken by the endpoint. We needed to observe the different steps in the process and see all the sections where the time taken could be reduced. We needed to see if it was really ES that was the culprit.

This is where Server-Timing API came into picture. Something that would break down the request into pieces and give us an idea about what’s expensive. The API allowed to pass the timings specific to a request from the backend to the browser. This meant that we could now see in the network requests, how the response times looked like, which part of the function took more time and so on.

To support this, we need to send a Server-Timing response header from the backend and time the methods that we needed. Something like this:

HTTP/1.1 200 OK

Server-Timing: db;dur=53, app;dur=47.2

The Server-Timing header can send three bits of information:

1. metric name
2. duration
3. description of the metric

In the above example:
db — metric name
dur=53 — the time in milliseconds the db metric took (This is for instance, the time to fetch some data from the Database)
app — another metric name and dur=47.2 being the time taken for the same.

The header can take multiple metrics separated by commas delivering great information and is super lightweight. Although it is recommended to keep the metric names as small as possible.

An example of what the returned Server-Timing header looks like:

db; dur=142.715967; desc="getRelationsForEntity",flt; 
dur=36.777609; desc="filterEntityListByAccess",pgn; 
dur=22.96549; desc="getPaginatedRelations",hyd; 
dur=0.64605; desc="hydrateRelations",ddp; 
dur=0.13311699999999999; desc="dedupeRelations",total; dur=721.388338

And this information then translates into a more visual and helpful view as a part of the network timings tab of the request OR available via the PerformanceServerTiming interface.

To access the three available properties, they correspond in the PerformanceServerTiming as:

• “name” -> PerformanceServerTiming.name
• “dur” -> PerformanceServerTiming.duration
• “desc” -> PerformanceServerTiming.description

Here’s the working draft with more information.

Timings tab to see server response timings

These are not available as a series of timings as such and just simple metrics and therefore, do not show in a waterfall pattern.

This means we can now tackle both the frontend performance problems and the backend slowness problems all in one place. No jumping to the logs to find a response was slow or heading backing again to the devtools to find the frontend problem.

We could also read the data now available using the navigation (PerformanceNavigationTiming) and resource timing (PerformanceResourceTiming) APIs and send it our analytic tools to create metrics/monitors.

const performanceObserver = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    log('Server Timing', entry.serverTiming);
  }
});

performanceObserver.observe({type: 'navigation', buffered: true});

Below is one of the examples from a method taking different amounts of time at different steps in the request. We will set up together the whole setting up and measuring the server response times later in this article.

Server response timings for a request endpoint in Browser devtools

This was especially helpful for the Elasticsearch example that I shared earlier as we needed to know which change was improving what part of the implementation and if it was moving us closer to improved response times. The Server-Timing header does not directly improve the timings but helps find out what could be worked towards.

The uniqueness of the API is that its super flexible and allows the server to communicate important information to the browser, even other than the server timings. It would be amazing to have this supported on Safari as well, which currently lacks the API support.

Coming back to how we came down to the API response in 400ms seconds as compared to 1.8s earlier

These are the two versions of the API. When we measured the performance with server-timings on the ES implementation after switching to the IDs search query on Elasticsearch, the performance seemed to be taking all the time, especially on this function searchRelationsForEntity .

Server timing indicating huge time taken by method searchRelationsForEntity

Now we knew this was to do with ES but where exactly. We timed specifics inside this method.

On measuring further, it was entitySearch.searchEntities that was taking up the most time so we needed to find a solution that would not go through all the ES documents at all to get the relation entities.

This is where changing the design made sense, and we moved to a graph table design with Dynamo, which would store all the relationships as mappings, and return all the entity IDs whether as direct relations or reverse.

Now we could choose if we wanted to search ES with all the IDs that we have or just batch get the entities from the db, and support pagination (this was an added advantage with ES) etc. separately.

Now, this complete dynamo approach saved us looking through all the indexed documents for the IDs that we need, and also cut some slack on Elasticsearch which would have this unnecessary load, eventually affecting other straightforward searches as well.

This is how the server timings looked with the DynamoDB approach:

Server timing indicating reduced times after switching to Dynamo to fetch entity relations

This was an amazing improvement, clearly visible with this significant performance information right in the browser devtools for us. This also ended up helping improve the code in general, to gain some more milliseconds by refactoring, and using performant ways to filter/hydrate/paginate things etc.We love to see the output of what’s in the works, and this was a great example to look at and feel a sense of achievement for.

Implementation of Server-Timing as a middleware for AWS Lambda

Now if you look up the existence of libraries supporting server-timing header, there are so many but most of them for Express and NodeJS.

• server-timing
• server-timings etc.

We needed to create one for the serverless use case which could be used for AWS Lambda, lets say to use with middy for example.

Step 1. Create a withServerTimings() middleware to use for AWS lambda

import middy from '@middy/core';
import * as Lambda from 'aws-lambda';

export const withServerTimings = () => {

  return {
    // add Server-Timing header to response
    after: (handler: middy.Request) => {
      const response = handler.response as Lambda.APIGatewayProxyStructuredResultV2;
      try {
        const headers: unknown[] = [];
        /**
          This is where the headers with different metrics are be saved
          and retrieved from
        */
        const timings = getServerTimingHeader(headers);
        response.headers = {
          ...response.headers,
          'Server-Timing': timings,
        };
      } catch (e) {
        Log.debug(`Middleware Error: Could not record server timings`, e);
      }
    },
  };
};

This creates a usable withServerTimings() middleware to then be used as:

Step 2. Apply the middleware to the request handler

return middy(handler)
  .use(...)
  .use(withCors())
  .use(withServerTimings())
  .use(...)

This already ensures that your backend will now be sending a Server-Timing header in the request endpoints.

Next step would be to actually time our operations and store them as a common timing variable which can then be sent to the getServerTimingHeader method for example.

Step 3. Implement timing for your methods

const times = new Map<string, Record<string, unknown>>();

// starting the timer on a method
const startTimer = (name: string, description?: string) => {
  times.set(name, {
    name,
    description: description || '',
    startTime: process.hrtime(),
  });
};

// ending the timer and recording the duration of the method
const endTimer = (name: string, description?: string) => {
  const timeObj = times.get(name);
  if (!timeObj) {
    return console.warn(`No such name ${name}`);
  }

  const duration = process.hrtime(timeObj.startTime as [number, number]);
  const value = duration[0] * 1e3 + duration[1] * 1e-6;

  timeObj.value = value;
  times.delete(name);
  return timeObj;
};

// finally setting the metric value received from endTimer object
const metric =
  typeof description !== 'string' || !description
    ? `${name}; dur=${dur ?? 0}`
    : `${name}; dur=${dur}; desc="${description}"`;

This would then be used as:

Step 4. Usage of timers of the methods

startTime('db', 'getRelationsForEntity');
const relations = await getRelationsForEntity({
  ...
});
endTime('db', 'getRelationsForEntity');
  .
  .
  .
startTime('pgn', 'getPaginatedRelations');
const { relations: paginatedRelations, hits } = getPaginatedRelations({
  ...
});

endTime('pgn', 'getPaginatedRelations');

And there we go, we will have the timed methods and the server headers looking like:

db; dur=139.99; desc="getRelationsForEntity",flt; dur=36.68; desc="filterEntityListByAccess",
pgn; dur=39.37; desc="getPaginatedRelations",hyd; dur=0.61;
desc="hydrateRelations",ddp; dur=0.130; desc="dedupeRelations",
total; dur=741.53

Visual representation of the server-timing headers

All the timing implementation and setting up the middleware can be a hassle, and should be a one-time job. I created a package out of this implementation called lambda-server-timing:

• lambda-server-timing (npmjs.com)

• GitHub – NishuGoel/lambda-server-timing (github.om)

• withServerTimings to use a middleware
• startTime to start measuring a function/set of functions
• endTime to end the timing of the function/set of functions and get the metric set.

import {withServerTimings, startTime, endTime } from 'lambda-server-timing';

Let’s time our functions down!

References

• Working draft
• MDN Server Timing
• Navigation Timing
• Resource Timing

I have come to realise a pattern in my past few years.

It happens to be a series of one type of things that I do, and I do a lot of that for the year. For example, 2018 was “writing blog posts” year, wrote more than 50 blog posts. 2019, “the reading books” years, read 36 or so books that year. 2021, again a lot of writing that year, 2022, “insane traveling” year.

TL;DR

Courtesy: Revolut

When I started this year, I was aware of one travel that was going to happen, for sure. It was the one that involved me moving to a different country to work. However, there was so much more in store than what I expected.
It felt like telling Covid that enough of your shit.
The point is, I ended up traveling to all the countries that I would think about traveling to, one day. And I wouldn’t have been writing this proudly, if it was not work-related. All of the travel was tech related. And what can be better than that?

Started the year with speaking at my favourite conference — ngIndia, and the energy was crazy since it was happening now after covid. Knowledgeable speakers, getting to know people.

I joined epilot beginning of this year which involved lots of learning and working with amazing people and in the best work culture one could have.
One important thing I realised was:

“A company is really doing something right, if even its engineers talk about the product (bigger picture) with as wide eyes and great passion as the founders.”

And I see this at epilot everyday and this inspires me so much to have such a culture at my own company, someday.
Like I mentioned in the beginning, this year was so much traveling and I am going to write small interesting instances from all my trips here, so I can come back later to this and smile.

Started with me traveling to the company summit in Germany which involved a week of fun with colleagues. This was when I got to know that. a beer CAN cost less than water sometimes if you are in Germany 🙂 Also saw, the team that parties crazy also achieves crazy.

And before it starts sounding like an epilot appreciation post, I should also mention the amazing time I had at the GDE summit in Leh Ladakh. All the Google developer experts coming together after covid, meeting in person, doing activities, playing games, was exactly what was needed after those two dreadful years.

September brought the wave of travel and never stopped until December. I moved to a new place here in Köln, and the next day I was traveling to Romania already for a web talk at Devfest Craiova about JavaScript Surprises. It was a one day event meeting amazing developers followed by a wonderful day out in the city.

Visited the Nicolae Romanescu Park, the largest natural park in Eastern Europe, apparently. I think the highlight of the Romania trip would be the Papanași that I had and the Sârbă dance we did at a get together dinner event.

This was followed by a surprise opportunity to visit the Firebase Summit in New York as an Angular GDE. And New York is a dreamy city until you have visited it at least once. This was great fun as well with fellow GDEs exploring all the touristy stuff — Times Square, Wall street, Helicopter View of the city, Brooklyn bridge, Statue of Liberty, Cruise tour, Summit One Vanderbilt etc. The highlight has to be the event venue which surrounded The Little Island.

Next was a talk at JavaScript Poland in Warsaw, which was a trip full of learning. Attending workshop on Qwik, learning about Partytown, Solid.js and reactivity conversation with Ryan, Streaming concept and Out of Order streaming, etc. The highlight would be meeting Dariusz after 3 years of meeting him for the first time in ngIndia and the beautiful JS Poland stage.

Had a work vacation planned in Vienna, Austria in November and boy, the city is beautiful. We stayed at a hostel called Jo and Joe which had a creative element to it. We walked around the city and I got to see the first snow of the season there as well. The highlight of the trip was really just enjoying, playing with colleagues, winning the Dart game, but cluelessly 🙂

Last but not the least was the trip to London for another Devfest event. This as quite unexpected as I had visa issues until two days before the travel, but ended up working out a day before, and how could I miss it then. Shared about JavaScript Memory Usage at Devfest London, again followed by a day out in the city, again some touristy stuff, but also great conversation with fellow speakers.

I happened to be in London during Christmas and that in itself is the best highlight, but there were so many. It snowed there making it even more beautiful, sitting in those London special red buses. Walking around the city and having important conversation with a friend was what made it all worth.

You know how when we realise a pattern, it is no more fun to still let it going. We, at least I like to see if the pattern can be broken and can something good come out of it. I think, I would really not want this year to be a series of one kind of thing.
Would really like it to be a mix of all kinds of things I like to, need to do, with the right amount of balance, consistency etc. Looking forward to writing, reading, creating more value this year. 🙂

Also, moving is quite an adventure. We introduce new music, food, culture, and new habits in general to our lives, sometimes language as well. Example being, 😀

Some talks:
– Devfest London
– JS Poland
– Devfest Craiova (recording)
– Devfest Bangalore
– ngIndia (recording)
– Web Directions — Global Scope

Some slides:
– microFrontends
– JavaScript Surprises
– Understanding JS memory Usage

Some writings:
– Breaking up long tasks — Perf Calender
– Dabbling at epilot
– Unraveling the Web
– Svelte-i18next

Some books read:
– The midnight library
– The T-shirts I love
– Thinking fast and Slow
– AI 2041
– The Bitcoin Standard
etc.

Prost! 🍻

This is the kind of experience, we as web engineers, cannot afford to give to the users. And there can be multiple reasons that could lead to this. If you regularly check the performance of your apps, you would have come across suggestions to code-split your application and reduce shipped JavaScript. In this post, we look at the causes of slow responsiveness of an application, how we could measure the slowness, and the approaches to improve it.

TL;DR
Certain tasks on a user interaction can be blocking, giving the user a broken experience, and resulting in a low INP metric score.

Some ways to optimise it could be:

• keep tasks small and dedicated to what they should do.
• use Chrome’s scheduler API to schedule tasks for browser
• yield important tasks to the main thread

Let us first understand why a user interaction could be slow. When a user interacts with an element on the browser, the user agent takes that as one task and queues it in the list of tasks for the browser. This one task could take more time or less depending on the type of operation needed to be performed. Some tasks take longer resulting in the main thread being busy performing that and leading to the the other tasks being blocked in the meantime.

Long tasks

A task is considered a long one if it goes beyond 50ms to execute and 50ms to respond, basically, more than 100ms. A website which doesn’t have long tasks, is expected to respond within 100ms with both the execution and response time on a user input, according to RAIL.

long task blocking user response

As per data from Web Almanac, it’s interesting to note that a median mobile page tends to have 19 long tasks.

Imagine the scenario where you click on an element and expect a response but the element doesn’t seem like it was clicked and doesn’t respond, so you click again. Those two clicks however, when the response completes, lead to undoing the action you wanted. Ever happened with you? Sounds frustrating, doesn’t it?

This is the kind of experience, we as web engineers, cannot afford to give to the users. And there can be multiple reasons that could lead to this. If you regularly check the performance of your apps, you would have come across suggestions to code-split your application and reduce shipped JavaScript. In this post, we look at the causes of slow responsiveness of an application, how we could measure the slowness, and the approaches to improve it.

TL;DR
Certain tasks on a user interaction can be blocking, giving the user a broken experience, and resulting in a low INP metric score.

Some ways to optimise it could be:

• keep tasks small and dedicated to what they should do.
• use Chrome’s scheduler API to schedule tasks for browser
• yield important tasks to the main thread

Let us first understand why a user interaction could be slow. When a user interacts with an element on the browser, the user agent takes that as one task and queues it in the list of tasks for the browser. This one task could take more time or less depending on the type of operation needed to be performed. Some tasks take longer resulting in the main thread being busy performing that and leading to the the other tasks being blocked in the meantime.

Long tasks

A task is considered a long one if it goes beyond 50ms to execute and 50ms to respond, basically, more than 100ms. A website which doesn’t have long tasks, is expected to respond within 100ms with both the execution and response time on a user input, according to RAIL.

long task blocking user response

As per data from Web Almanac, it’s interesting to note that a median mobile page tends to have 19 long tasks.

This is where INP comes into picture. INP is a responsiveness metric, stands for Interaction to next paint, which measures the time from when the user interacted with the page until when the next frame was actually visible on the page. A good INP measure is when the time between interaction and the feedback as a result of that interaction is the least.
An interaction could mean events like tap, click, keyUp, pointerdown etc.

An interesting read here on how’s INP actually measured .

INP is calculated by observing all the interactions made with a page. The chosen value is then a percentile of those interactions. A formula is then used to choose a high percentile value of those interactions.

The way we put web vital measures in a Good, Needs Improvement, Bad buckets, here is what a good and poor looks like for the INP metric.

To calculate the overall page responsiveness, an INP measure is calculated when the user navigates away from a page, and this comprises of all interactions like mouse events, keyboard events, touch/tap on a touchscreen device.

FID / INP?

I remember analysing my websites, and always being satisfied with at least one metric of the three core web vitals 😀 — the First Input Delay. Would up my ego a bit that I am doing something good for sure. But if we look at the bigger picture, it is noticed that most websites have a good FID value, even when the responsiveness experience isn’t that great.
And to address this, INP with a better measurement approach has been introduced. INP doesn’t just measure the first interaction but all interactions on a page, unlike FID. Also, as the name says, INP measures the time until the next paint happens, unlike FID, that measures the time only until the processing happens.

If we look at the INP scores on mobile devices, 55% of websites have a good INP score, and 36% need improvement. However, in case of FID, 92% of websites having “good” FID. So, a better responsiveness measure with INP already brought down the “good” score.

It is also interesting to see that INP score tends to be inversely proportional to the popularity of a website, as per data from the Performance chapter in web almanac.

With a better approach to measure the responsiveness of a website with INP, it would make sense for developers to really start considering this metric as an important performance check, also with Google considering to replace FID with INP as one of the core web vitals.

When looking at the mobile experiences by site rank, 52% of top 1K websites fall in the good category with FID metric, however only 20% pass with good status with INP as a metric, and this is a huge difference hinting towards the need to really improve the overall responsiveness situation, long tasks being one of the things to solve.

Can I track my website’s INP value?

One cool way to track our website’s interaction to next paint measure with real data from users is using the web-vitals library.

The library offers the onINP method that returns the type of event along with the INP value.

We could record this data and then receive it via some API or analytics provider to be able to understand the reasons behind the INP measure based on the real usage from field.

Something like this: 

import {onINP} from 'web-vitals';
onINP(({name, value, attribution}) => {
  console.log(name)
  console.log(value)
  console.log(attribution.eventType)
});

This gives us the measure of our INP and helps us understand further how we this interaction delay be improved.

This is how the general INP score looked like on websites as per the data from Web almanac.

How do you find a long task in your application?

By now, we would know how the INP value is doing, and the next step right away would be to profile our app and see the improvement areas. To do this, we go straight to the performance profiler and identify the long tasks (ones exceeding the threshold of 50ms).

As we can see, there are many long tasks leading to slower input response to the user, one such having a delay of 676ms.

And as we dive deeper, it gives us some idea about the operations causing this to go up to 600ms here:

Let us try to optimise one such long task now. There can be different approaches to improve long tasks, with one being breaking up your long tasks into smaller dedicated tasks.

For example, the long task from above which took 600ms, was a blocking task for UI related updates, and thus would result in a broken experience for the users since they’d see a delayed response. What if the task still does exactly what it does but it is broken into multiple tasks with small operations, meaning:

• Task 1 — Only for querySelector step
• Task 2 — parse
• Task 3 — getOperations
• Task 4 — removeChild
  and so on.

This would then allow us to inform the browser to NOT delay the removeChild step because of the time-taking parse step which takes around 5.8ms only for some parsing. And thus, allowing the browser to give a faster visual feedback to the user without having to wait for the multiple expensive steps before that.

Informing the browser to perform a task which is more urgent than others can be called “yielding to the main thread”. One way to do that is using a Promise with a call to a setTimeout function:

function yieldToMain () {
  return new Promise(resolve => {
    setTimeout(resolve, 0);
  });
}

This setTimeout call will then ensure to run the dedicated step as a separate task prior to the other queued-up tasks. It depends then, how one would want to prioritise their tasks, and when yield exactly.
In our long task example, it would make sense to yield to the main thread after we expect some user visual as a response. For instance, showing a loading spinner before the parse step, which would mean that now we already have some response for the user in 271ms instead of the 600ms.
This is already something better as compared to before.

One more use case could be to only wanting to yield to the main thread and perform a crucial task, when we see that there is an action that the user is waiting for a response from. In this case, it would make sense to yield to the main thread only after knowing that some input is pending.
And that’s where the API isInputPending() comes to the rescue.

isInputPending

Facebook contributed the isInputPending API to chromium based browsers which tells the browser if there is a user input that is waiting for a response. This is useful to be able to respond quicker to such tasks that need a visual feedback.

Read more here 👇🏻

Breaking up long tasks

Web performance calendar 2022

So, turns out there can be multiple reasons for this to occur:

1. Applying browser translation on non-wrapped elements.
2. Conditionally rendering elements inside React Fragments.

Applying browser translation on non-wrapped elements

<DropDownMenu
  id="create-menu"
  menuItems={items.map(item => ({
    name: item.key,
    label: i18n.t(item.slug), // This is where a non-wrapped element is getting translated causing the error
  })}
>

{...}

</DropDownMenu>

Solution:
Passing label as a JSX element instead of a string.

<DropDownMenu
  id="create-menu"
  menuItems={items.map(item => ({
    name: item.key,
    label: <span>{i18n.t(item.slug)}</span>, // This is where a non-wrapped element is getting translated causing the error
  })}
>

{...}

</DropDownMenu>

Conditionally rendering elements inside React Fragments

We use fragments in React to avoid adding extra nodes. These fragments when used conditionally, for example for a use case like:

<Fragment>
   {!isThisSituation ?  
        <span>Display this</span>
          :    
        <Icon
            color="secondary"
            fontSize={12}
            name="this-icon"
        />
    }
</Fragment>

result in React trying to remove the conditionally rendered node from the parent that is a fragment. However, we know Fragment doesn’t really act as an extra node.

Solution:

When conditionally rendering nodes, wrap the children inside a legit node.

<div>
   {!isThisSituation ?  
        <span>Display this</span>
          :    
        <Icon
            color="secondary"
            fontSize={12}
            name="this-icon"
        />
    }
</div>

Also throws the error for a scenario like this:

// throws the error if:

<div>
  {condition && 'text'} // solution: needs to be wrapped in an element
  <span>Some other text</span>
</div>

OR

<div>
  {condition && <span>Text</span>}
  Welcome // solution: needs to be wrapped in an element
</div>

Let’s look at the tech stack involved.

Micro services

The main epilot app is built using microfrontends with each entity as its own application. There is a login application that takes care of the user login state (built using React + TypeScript), a topbar application that has features like Global search, notifications (React + TypeScript), a sidebar application that navigates the user to different entities based on user permissions (Svelte + TypeScript + Tailwind), and so on.

The component library of the product is based on material UI, built with React.

@epilot/base-elements

Due to the open source experience of many engineers in the team, most of the non-confidential stuff is kept public on GitHub and npm.
One of the very interesting things I got to do to ramp up was to build a part of the product as a Svelte MFE. With most of the existing MFEs built using react, this was quite interesting and challenging to do.

We use single-spa framework to build our microfrontends and use different frameworks per the ease and knowledge of an engineer. To make the development journey easier for a new developer starting up with microfrontends, we have a mfe-app-generator package that creates a boilerplate application with the required config.

@epilot360/mfe-app-generator

Due to performance reasons, one of the goals was to switch to rollup for bundling all our applications, however, we use webpack due to rollup’s known issues with aws-sdk.

We use serverless microservices architecture exposed via APIs. Most of the backend services are written in TypeScript using AWS lambda functions, APIs, Cloudfront distribution etc. More on that here.

With Svelte added to our microfrontend stack, we plan to further diversify the frameworks to be used, ensuring to not lose focus on the performance aspect of the application. Here’s a list of support frameworks by single-spa.

I also plan to share our journey from a high LCP (largest contentful paint) to a much reduced one, but I’d keep it for another blog post.

When working with custom implementation and bundling, one of the challenging tasks is to configure your bundler and ensure it is consistent throughout all the applications (MFEs in this case). We have defined a @epilot360/webpack-config-epilot360 to base one’s webpack config on top of the existing basic config defined.

@epilot360/webpack-config-epilot360

I will share more on the webpack configuration for microfrontends and cache invalidation strategies in another post but cache invalidation is definitely my major ongoing focus area.

Epilot SDK

All public epilot services are available through an epilot-sdk (written in JS/TS) that allows developers to interact with epilot APIs. Internally as well, the recommended way to use the services or APIs using directly the SDK instead of going the granular way.

For e.g.,

import { getClient } from 'epilot-sdk/entity-client'

Want to scratch your itch? Find it here:

@epilot/epilot-sdk

Communicating between microfrontends

One of the interesting challenges I recently faced was to communicate between two applications without any shared library. One was the sidebar built using Svelte and another the topbar built with React. Now we usually tend to avoid any such communication between our MFEs but this was a state clear event to happen before a routing change in any of the two MFEs.

I used single-spa’s custom events to do this. The idea was to clear the selected state of an icon in the topbar when an icon in the sidebar was selected, and vice versa.
To do this, I used the single-spa:before-routing-event custom event to check if the active URL in the current application was undefined, we could clear the state, if however, the URL existed, that becomes the active state. 
Checking the URL history without communicating between two applications was not possible due to two separate routers for each application. Svelte app using the svelte-routing package where with React, the react-router .

Read more about the single spa custom events here.

Here’s a detailed view of the tech stack used at epilot:

https://docs.epilot.io/techradar/

I have shared more info on my journey to epilot as a full-stack developer in my Spotify Podcast here:

Spotify podcast

Interested to know more about how we do things at epilot? Reach out here: twitter.com/@TheNishuGoel

So I took up a bigger challenge for myself (yes, open-source, of course) and with a better read for y’all. Yes, it’s a lot of content.

So, I authored the JavaScript chapter for the Web Almanac 2021. To keep you up to speed, The Web Almanac is:

HTTP Archive’s annual state of the web report that combines the raw stats and trends of the HTTP Archive. It is backed by real data and trusted web experts. The 2021 edition is comprised of 24 chapters spanning aspects of page content, user experience, publishing, and distribution.

In simple terms, it shares a lot of statistics around what is being used, adopted, preferred on the web by the developer community. How are the features being used and if the way they are used, serve the sole idea of the introduction of that feature.

My participation started while I was exploring GitHub activities, and ended up coming across at the Http Archive’s JavaScript repo for 2021. I volunteered to author the chapter but ended up becoming the content team lead whose responsibility was not just analysing the data from the 8 million websites crawled and writing about it, BUT many other steps.

I’ll be honest that I was overwhelmed with the amount of work to do and get the chapter in its shape, but it was challenging and basically, lots of data (power)!

The phases (starting June 2021) in the complete chapter building involved:

1. Planning the content and preparing an outline.
2. Gathering the data and writing custom metrics (in JavaScript). These were run during the crawl after the page was rendered in the browser.
3. July 1–31 is when the crawl took place.
4. Next was to analyse, query the data and save the results to a sheet which is the source of truth for all the content in the chapter. This step also involved ensuring that our custom metrics ran properly and output was expected (I have an interesting anecdote to share about this later).
5. Now started the draft writing phase based on the understanding formed after the data output. This was, as it seems, the most involving task in the whole process. Not only because there was loads to write but also because, you need to analyse as to why, for example, there is increase/decrease in the usage of one particular JavaScript feature.
6. Nope, doesn’t end there. This is where the review process starts where your reviewers help you frame sentences better for a common and easy understanding. They help you with forming the overall chapter better both content and usage wise.
7. Editing — This step was more about language improvements and obvious mistakes that should be removed.
8. To the moon. 🚀

In the next few paragraphs, I will share about the learnings and challenges at each step. The purpose of doing this is 1) to reflect how helpful and growing of an experience this was personally, 2) to help anybody who is considering writing the chapter in the coming years, 3) to point you the great resource (all JavaScript and more), if you haven’t already come across this ocean of information.

• The outline planning stage was super fun, full of research where we were all creative and thinking of putting in the best possible statistics and information. This involved the newer JavaScript features based on the TC39 finished proposals to the community. My fellow contributors helped me in the process of preparing the outline and brainstorming the content. Pankaj Parkar was very helpful in this step suggesting ideas and possibilities.
• Writing custom metrics was based on the author’s understanding of what they plan to deliver through the chapter and what is the data that they need from the websites to showcase that. This involved thinking ahead on all what could be available from running queries on data, or something that needed to be captured after the page render.

For example, To check if the script tags use the async/defer attribute, there were two ways — 1) check initial page and look for scripts with these attributes — This did not require running a custom metric and could be simply attained by running a query on summary response table in the database.
2) To write a custom metric to get the no. of script tags with async or defer attribute and then run a query to get this percentage.

https://github.com/HTTPArchive/legacy.httparchive.org/blob/master/custom_metrics/javascript.js#L62

The results were different from each other. You’d wonder why:

“This difference shows that many pages update these attributes dynamically after the document has already been loaded. For example, one website was found to include the following script….”

We used web page test to run our custom metrics and ensure that’s what we are looking for from the crawl.

For example:

WebPageTest to run our script tag custom metric

One challenge faced with the custom metrics was specifically with the web component custom metric where we made a mistake in the code ending up not getting correct results from the July Crawl. We fixed this for the next crawl and the results in the chapter were, thus, delivered from the September crawl for this particular statistic.

See the note here:

https://github.com/HTTPArchive/almanac.httparchive.org/blob/main/sql/2021/javascript/web_components_specs.sql

• The next was analysing the crawl data. The queries were supposed to be written using BigQuery which was a good learning activity. We wrote the queries and saved the output to the result sheet. The analysts helped us with saving the data and creating the visuals for that. All the interactive charts and statistics.

Following queries were used to gather the data for the chapter.

https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/sql/2021/javascript

The charts created from the query data were published both interactively and as images. Finally wrote the chapter in a doc, converted to markdown, added the figures to the chapter in a render-able format.

https://github.com/HTTPArchive/almanac.httparchive.org/blob/main/src/content/en/2021/javascript.md

The conversion of visuals to a format that adds caption, description, and interactive charts was definitely a lot of task after the writing the chapter but definitely worth it.

https://almanac.httparchive.org/en/2021/javascript#javascript-requests-per-page

The chapter finally launched on December 1after all the reviews, edits added, accommodated, and this definitely felt like an achievement.

The chapter was welcomed very well and got a lot of people talking. Especially with the following statistic which show the usage of jQuery still at 80%.

Credits: https://almanac.httparchive.org/static/images/2021/javascript/js-libs-frameworks.png

And the usage of async/defer statistic:
JavaScript (Web Almanac 2021)Pages using async/ defer / neither

Credits: https://docs.google.com/spreadsheets/d/1zU9rHpI3nC6jTz3xgN6w13afW7x34xAKBh2IPH-lVxk/edit#gid=2038121216

In the first week of publishing, it became the most popular chapter of the Web Almanac with more than 2K views already. You can view the 2021 Web Almanac analytics here:

https://datastudio.google.com/s/gNBIUwcc8LE

Some amazing statistics about this year’s Web Almanac are:

7 months of work
736 queries and 48,218 lines of SQL
24 chapters
112 contributors
8M websites tested and 40 TB of data analysed
Over 700 pages when saved to PDF

You can download this as an ebook too.

You can already sign up to contribute to the next year’s report here. This could be as an author, reviewer, editor, and/or as an analyst.

https://forms.gle/Bb2ZZj6Yw5RnJHbj8

This is about what I was doing the past 4 months and what comes next is more detailed blog posts from me on the Web in general.

Follow me on Twitter for more details and staying up-to-date.
OR
If you are not a Medium fan (maybe because of the paywall, or because of the hassle of going to incognito to open my posts),
I have all the content also served on my personal website, which is absolutely free, no paywall, no ads.

https://unravelweb.dev/

Thank you!

For the #javascript 2021 chapter of web-almanac, I’d like to know from the #javascript developers here if you’d be interested to see any special metrics in the report?

Metrics like: I, as an Angular dev, would be very interested to know how many applications use SSR or as a JavaScript dev, what differential loading method is used in majority JS apps.

No clue what I am talking about? Refer to this and share your thoughts:

https://github.com/HTTPArchive/almanac.httparchive.org/issues/2141

Do let me know in the comments.

The conference revolves around Scalable infrastructure and architecture, Maintainable Code, Automated Testing etc.

Would like to be a part?
Use my speaker code “Speaker”to get 50% off of the current ticket prices or simply click this link.

Date: August 26–27, 2021

See you there 🙂