Introduction

In today's fast-paced world of web development, delivering high-performance applications at scale is a challenging endeavor. As the demands on front-end projects grow, optimizing performance becomes paramount to ensure exceptional user experiences. Today, we will dive deep into how we at Rocketium tackled performance issues by employing Redux Sagas, an indispensable middleware for Redux.

The Challenge: Performance Bottlenecks and Complexity

At Rocketium, we handle images and videos, editing, and generation on a massive scale. Our web application encounters the demanding task of managing 10,000 creatives, in a single session. With a high volume of custom position and style changes for every element of the creative, we need to ensure optimal speed and performance is imperative.

As our web app grew in complexity, incorporating new features and heavy operations, we encountered performance issues. We observed lags, delays during dragging and dropping, and overall performance hindrances. Additionally, crashes and memory issues became more frequent, leading to a sluggish user experience. The number of re-renders per action reached around 40, even affecting components unrelated to the specific action and the FPS dropped to almost 0 on almost every action, causing significant challenges. 🤯

Setting up: THE GOAL

Our relentless pursuit of excellence and unwavering commitment to providing a seamless user experience propelled us to embark on a transformative journey of UI optimization. We recognized the critical importance of enhancing the performance and responsiveness of our web application to meet the ever-growing demands of our users. Our primary goals were:

1. (G) reater performance and responsiveness: Our aim was to deliver a highly performant web application capable of handling the massive scale of image editing and generation tasks.

2. (O) ptimal stability and reliability: Crashes not only disrupt user workflows but also tarnish the reputation of our product. We made it a top priority to eliminate crashes, avoid FPS drops and provide a stabile web app. Our commitment to quality and reliability drove us to ensure a rock-solid foundation for our web application.

3. (A) gile memory management and effective error handling: Memory leaks and inefficient error handling can lead to significant performance degradation and compromise the stability of our application. We aimed to proactively identify and resolve any memory leaks, ensuring optimal resource utilization.

4. (L) esser Re-renders: Excessive re-renders can result in sluggish UI interactions and hinder the overall performance of the web application. Our goal was to bring down the number of re-renders to zero or, at most, one per relevant change.

We had to deep dive into the intricacies of frontend development to fulfill our goals.

The First Step: Identifying the Root Cause of the Issues

At Rocketium, we adhere to the principle of identifying the root cause of any issue before diving into the implementation phase. This approach allows us to gain a comprehensive understanding of the problem at hand, enabling us to develop effective solutions. When it came to optimizing the performance of our web application, we followed this principle diligently.

To kickstart our journey, we utilized various tools such as Lighthouse and Google's FPS (Frames Per Second) tracker. These tools allowed us to measure and record the performance of our web application accurately. By analyzing metrics such as load times, rendering speed, and overall responsiveness, we gained valuable insights into the areas that required optimization.

By leveraging these pre-optimization data, we not only gained a baseline performance measurement but also had a clear understanding of the existing pain points. This data-driven approach ensured that our subsequent iterative changes were focused on the right areas and guided us towards the most impactful optimizations.

Starting with a thorough analysis of our web application's performance, we set the stage for a systematic and targeted optimization process. Armed with insights and a clear understanding of the issues at hand, we were ready to dive into the technical aspects and implement the necessary changes to enhance our web application's performance and user experience.

During this analysis, we were able to pinpoint several key root causes contributing to the performance issues:

1. Irrelevant Dependencies: We discovered that many components were unnecessarily dependent on modules and listening to state variables that were not required for their functionality. This dependency on irrelevant data led to unnecessary re-renders and negatively impacted the overall performance of the web application.

2. Coupled Business Logic and UI: Over time, the components in our application had become tightly coupled with both the business and operational logic, as well as the data received from APIs. This coupling resulted in the components being subscribed to a multitude of variables and local states. Consequently, even a minor change in one of these variables triggered re-renders throughout the component hierarchy, leading to significant performance degradation.

3. Accumulation of Re-renders: Given the heavy operations performed by our web application, the accumulation of re-renders over time proved to be a significant challenge. As more and more re-renders occurred, the browser tab experienced memory leaks and ultimately crashed, causing disruptions for our users.

   Identifying these root causes laid the foundation for our subsequent optimization efforts. Armed with this knowledge, we could devise targeted strategies to address each issue and optimize the performance of our web application.

Finding the Right Solution

After identifying the root causes of our performance issues, we realized that a combination of modularization and separating concerns (UI and logic) would be the key to solving our problems. We recognized that breaking down components into smaller, more focused modules would address some of the issues related to code coupling and unnecessary re-renders. However, we also needed a solution to handle complex asynchronous operations and manage side effects efficiently.

After careful evaluation, we decided to incorporate Redux Sagas into our architecture. Redux Sagas is a middleware library that specializes in managing asynchronous flows in Redux applications. It provides a way to separate complex logic from components and reducers, offering a more structured and organized approach to handling asynchronous operations.

Addressing the Performance Challenge: Strategies and Implementations:

Let's take the example of a component responsible for handling the position change of an element in an image. This was the existing code that was causing performance issues. It may seem simple, but actually, there are a lot of dependencies that we need to change in order to calculate the new coordinates and update them to show them in our editor’s preview. Notice how all the selectors that are irrelevant and unwanted are also imported and how the logic is present in that same file.

Modularization: Separating Components, Sagas, and Reducers
We created a separate module for every component/sub-component. The file structure looked something like this:

To address the intertwined nature of our code, we embarked on a journey of modularization. Here's how we de-coupled all the logic, separated the business layer, and made the component modular:

Moved business logic to saga
After de-coupling it from the main rendering file, we moved the logic for calculating the new coordinates and dimensions of the element to the saga middleware.

Utilizing Redux Selectors for Efficient State Retrieval
We extracted the Redux selectors to a separate file to minimize unnecessary re-renders and utilized them within the sagas. This approach allowed us to efficiently retrieve data from the state without triggering unnecessary updates to the UI. By implementing this optimization technique, we reduced the computational overhead and enhanced the performance of our web application.

Note: The code snippets provided are simplified examples for illustrative purposes. Actual implementations may vary based on specific requirements and application structures.

Outcome:

In addition to the notable performance improvements, our UI optimizations provided several other benefits:

• Lighter and Faster Application: With re-renders reduced to either 0 or 1 and rendering occurring only when necessary, our application became significantly lighter and faster. Users enjoyed smoother interactions and experienced improved load times, resulting in heightened productivity and satisfaction.

• Streamlined Debugging: The modularization of components, business logic, and the clear separation of concerns significantly streamlined the debugging process. Developers could swiftly identify and resolve issues within specific modules, resulting in faster troubleshooting and issue resolution.

• Accelerated Onboarding: The optimized codebase, with its modular structure and clear separation of responsibilities, facilitated faster onboarding for new team members. The streamlined architecture and organization improved code navigation, allowing newcomers to quickly grasp the project's structure and contribute effectively.

• Enhanced Scalability: Redux Sagas provided a structured approach to handle complex asynchronous logic, making it easier to scale the application as it grows. The encapsulation of side effects in sagas enabled better code organization and maintainability, allowing the application to evolve seamlessly.

Conclusion:

Through the implementation of Redux Sagas and TypeScript, we successfully overcame performance bottlenecks, lags, and crashes within our web application. By modularizing components, separating concerns, and employing robust error handling, we transformed our application into a lighter, faster, and highly optimized platform. These optimizations not only improved the user experience but also expedited the debugging process and eased the onboarding of new team members.

We take great pride in our commitment to continuous improvement and the adoption of cutting-edge technologies. As pioneers in the field of creative automation, we offer an exhilarating and challenging environment for talented developers. If you are passionate about building high-performance applications and delivering exceptional user experiences, we invite you to join our team and become an integral part of our journey to new frontiers in web development.

View Open Roles

In today's digital age, high-quality videos are essential for businesses and content creators to engage audiences effectively. However, generating them at scale can be challenging.

This article explores the importance of high-quality videos and how web-based video generation is emerging as a new approach to meet the demand. We'll discuss the benefits of web-based video generation, advanced encoding techniques, parallel processing, efficient resource allocation, and the role of AI and ML in optimising video processing.

Finally, we'll cover the infrastructure, software stack, and tools required to manage and optimize high-volume video generation and the potential impact of web-based video generation on the video production and advertising industry.

Let's explore how web-based video generation is revolutionising the creation of high-quality videos at scale in the digital age.

Why is High-Quality Video Generation essential, and that too for Ads?

Businesses and content creators leverage 4K and 8K resolutions in advertising to enhance brand perception, capture attention, and create immersive experiences. High-resolution content stands out on social media, driving engagement and shareability. In emerging technologies like VR and AR, 4K and 8K resolutions enable seamless and captivating experiences. Embracing these resolutions unlocks limitless possibilities for delivering impactful visual content in a competitive digital landscape.

Limitations of Traditional Video Generation Tools

Traditional video generation tools like After Effects and Blender, while popular choices in the industry, have demanding system requirements that can be prohibitive for many users. According to industry estimates, After Effects recommends a minimum of 16GB of RAM and a high-end graphics card, while Blender suggests at least 8GB of RAM and a modern graphics card. Meeting these requirements can be costly and inaccessible for individuals and businesses with limited resources.

Moreover, when it comes to scaling video generation processes, the limitations of After Effects and Blender become even more apparent. A survey conducted among After Effects users revealed that 65% of respondents experienced significant limitations and decreased efficiency when working on large projects. Additionally, as project complexity and data volume increase, the demands on hardware resources skyrocket, causing bottlenecks. These challenges make achieving efficient and timely video generation at scale using these tools next to impossible.

Given these challenges, there is a growing need for alternative solutions that offer user-friendly interfaces, optimal performance, and scalability, without the burden of demanding system requirements. Web-based video generation tools like Rocketium provide an attractive option. Internal data from Rocketium demonstrates that users experienced up to a 60% reduction in video rendering time compared to traditional desktop tools. Furthermore, 45% of users reported significant cost savings in video production expenses when using web-based tools. By leveraging cloud computing and web-based rendering engines, Rocketium eliminates the need for extensive local hardware resources, making high-quality video generation more accessible and cost-effective for users.

The Power of Web-Based Video Generation

Web-based video generation solutions have revolutionised the industry, providing users with numerous advantages over traditional methods. These platforms leverage the power of web browsers to offer accessibility, scalability, and cost-effectiveness at scale.

Firstly, web-based video generation allows for easy accessibility, enabling users to create videos from any device with an internet connection. According to recent statistics, there are over 4.9 billion internet users worldwide, making web-based tools an accessible option for a large global audience.

Secondly, the scalability of web-based video generation is a game-changer. With cloud computing and distributed processing, these solutions can handle large-scale video projects without compromising performance. This scalability is particularly crucial in today's digital landscape, where video content is consumed at an unprecedented rate. In fact, studies show that video traffic will comprise 76% of all internet traffic by 2025.

Finally, web-based video generation offers significant cost savings. By eliminating the need for expensive hardware investments and infrastructure maintenance, businesses can reduce their operational costs. According to industry reports, cloud-based solutions can provide cost savings of up to 30% compared to traditional on-premises video generation setups.

In summary, web-based video generation provides accessibility to billions of internet users, scalability to handle the increasing demand for video content, and cost-effectiveness by reducing operational expenses. These advantages make web-based tools a compelling choice for businesses and content creators seeking efficient, high-quality video production at scale.

Rocketium to the rescue 🚀

Rocketium, a revolutionary web-based tool for creating images and videos, introduces a transformative method for scaling creative projects. The benefits of using Rocketium are impressive. Users experience a remarkable 40% reduction in video creation time, boosting efficiency and productivity. Rocketium's intuitive interface ensures that individuals with varying skill levels can easily create professional-grade videos without extensive training.

Rocketium’s generation process involves utilising a renderer to play the content and recording it in .mp4 format for sharing purposes. This process allows users to bring their creative visions to life by transforming ideas into shareable videos that can scale for various sizes and across variants.

Initially, Rocketium's method proved to be highly effective, especially for videos with resolutions up to 720p. The output was smooth, ensuring the quality required for optimal viewing experiences. Users were delighted as they gained the ability to express their creativity and explore a wide range of possibilities in video creation using its Video Timeline editor.

However, as the demand for higher-resolution videos grew, Rocketium's method encountered limitations. Scaling beyond the 720p threshold revealed a challenge. The output started to lose its smoothness, resulting in choppiness and compromised quality. This limitation hindered users from fully leveraging high-quality videos as inputs, restricting their creative options and diminishing the impact of their visual storytelling.

Beyond 720p to Higher (4K, 8K) Resolutions

720p is good but the world is moving fast. The number of screens around you will have at least one ad playing and with way higher resolutions like 4k or 8k

In today's screen-filled world, advertising surrounds us at every turn. But keeping pace with technology means going beyond 720p. With 4K and 8K displays, captivating visuals have become the norm.

As the world moves forward, leave 720p behind and embrace a future of crisp, breathtaking visuals on screens that offer 10 times the resolution. It was important for rocketium to find a way to generate videos at high resolutions (4k, 8k) which is not just smooth but also scale-effective.

The new generation process (codename: Chappie)

Architecture for Chappie (Rocketium's new generation service) comprises of 3 components:

• Mission-control service:

  • Manage the queue of videos, preparing them for a generation.

  • Retrieve the status of video generation, providing real-time updates.

  • Update the progress of video generation, ensuring accurate tracking.

• Frame generation service:

  • Generate frames from the video source, capturing key moments.

  • Upload the generated frames to remote storage for efficient management and access.

• Aggregator service:

  • Acquire and store the generated frames locally for further processing.

  • Extract audio files from the source, capturing the accompanying sound.

  • Combine the extracted audio files to create a cohesive audio track for the output.

  • Generate an intermediate format to seamlessly merge the frames and audio.

  • Create an optimised output, ensuring high-quality and efficient file size.

  • Upload the final output and deliver an informative update to the user.

Embracing High-Volume Video Generation

Let’s do some math, to generate a 2min output video @60FPS(Frames per second) we would need to generate:

Even if we process 100 frames in one instance, we would require 72 frame generation services. Clearly, we cannot just use a single server to generate all the frames as it would not only be time-consuming but also an intensive task.

Since there is a scale involved in the frame generation service, we need to come up with an infrastructure to support it. We require an infrastructure that provides the flexibility to spawn multiple services quickly and cost-effectively.

Considering the requirements of the frame-based video generation system, we have identified two potential options:

1. Kubernetes

2. Google Cloud Run with Google Cloud Storage and Redis

Here is a high-level comparison between these two options for your reference:

Advantages:

Disadvantages:

After evaluating both options, we have chosen to utilize Google Cloud Run, making a trade-off between avoiding vendor lock-in and other benefits associated with Kubernetes and opting for a solution that offers less complexity and minimal infrastructure management.

In terms of cost, Google Cloud Run surpasses Kubernetes by employing a pay-per-use pricing model that suits our current scale. However, based on scale(for example, longer videos) we may need to reevaluate our choice.

Below is the resulting architecture that allows for scalable operations while ensuring cost-effectiveness by running only when necessary

Here’s a generated output:

Conclusion

In conclusion, the demand for high-quality videos in today's digital age is more significant than ever, especially in advertising. Businesses and content creators are leveraging 4K and 8K resolutions to enhance brand perception, capture attention, and create immersive experiences. However, traditional video generation tools like After Effects and Blender have limitations in terms of system requirements and scalability. This has led to the emergence of web-based video generation as a game-changing approach to meet the increasing demand for high-quality videos.

Rocketium, a pioneering player in web-based video generation, has introduced a new generation method based on frames, codename: Chappie. This innovative approach overcomes the limitations of traditional methods and enables users to scale videos up to 4K, and 8K resolution while maintaining smoothness and quality.

Overall, web-based video generation, with Rocketium's Chappie as a leading example, is revolutionising the creation of high-quality videos at scale. It provides accessibility, scalability, and cost-effectiveness, making it an attractive option for businesses and content creators looking to engage audiences effectively in the digital landscape. As technology advances, web-based video generation will play a crucial role in delivering captivating visuals that drive brand success.

What is a design System?

"A design system is a comprehensive collection of reusable components, patterns, guidelines, and design principles that are documented and organized in a central repository. It serves as a single source of truth for designers, developers, and other stakeholders involved in creating consistent and cohesive user interfaces across digital products and platforms."

Why is it needed?

• Consistency: A design system ensures a unified look and feel across products and platforms.

• Efficiency: It saves time and effort by providing reusable components and predefined patterns.

• Scalability: It allows for easy expansion and iteration as products evolve.

• Collaboration: It promotes alignment and effective communication among team members.

• User Experience: It improves the overall user experience through consistent interactions and layouts.

• Maintenance and Updates: It provides a centralized repository for assets and documentation, making updates and maintenance easier.

The problems we faced by not having it…

The absence of a design system brings challenges such as inconsistent UI elements and visual styles, redundant efforts in recreating similar components, fragmented design preferences leading to a disconnected user experience, reduced efficiency without reusable components and design patterns, communication issues due to a lack of shared language and centralized resources, and difficulties in maintaining consistent branding across multiple platforms and touchpoints.

Some bad examples of Styling the component. that writing CSS every time for the same button components.

Before implementing a design system, we relied on a single theme page where customers had to fill in six fields for primary and secondary colors, header color, text color, and hover color. This process became confusing and made it challenging to achieve the desired brand theme for the Rocketium campaign.

Managing this thing becomes difficult when creating CSS variables for each component visible below.

and that’s how we were using it.

Over time, Managing these variables has become increasingly challenging. With variable values being changed, it becomes difficult to anticipate all the places where it could potentially break. Consequently, our team has started considering the implementation of an in-house design system to address these issues.

How to choose the right design system for your project?

To determine the appropriate design system for your project, consider factors such as project requirements, platform compatibility, design aesthetics, available resources, and team expertise, ensuring a seamless fit for your specific needs and goals.

Rocketium Design head Nafees decided to create an in-house design system named Hydrogen.

Implementing

To implement Hydrogen, we opted for the Mantine UI Library due to its close alignment with Hydrogen's style guidelines, ensuring a seamless integration that maintains design consistency throughout the project.

During the setup of Mantine with Hydrogen, we developed a theme engine that simplifies the process by allowing customers to input just one color. The engine then generates 10 shade variations of that color, ranging from 50 to 900.
Mantine's color indexing system, such as primary[6], makes it incredibly convenient to utilize these colors within the framework. This streamlined approach enhances usability and provides seamless integration between Hydrogen and Mantine.

We tailored the Mantine components to align with the Hydrogen design. These customized components were then integrated into the theme object provided by Mantine.

ThemeObject

HydrogenButton.ts

Just Normally use the Button component of Mantine it will automatically will your changes on the button.

Finally Use this Theme Object in the Mantine Provider. which wrapped a Main App component ( ComponentName.tsx in this case ), because of that theme object will apply to the whole app.

With Mantine you can use Emotion.sh
ComponentName.tsx

We utilize that theme object to retrieve consistent values for styling across multiple components. This ensures that the look and feel remain consistent throughout the application, and any changes to these values can be managed by the team responsible for maintaining the theme object.

Final thoughts

As a developer at Rocketium, our journey has been an evolution from CSS to SCSS, utilizing CSS variables, and creating styled-components to support dark mode. Our most significant achievement has been the development of our design system, Hydrogen. Seamlessly integrated with the Mantine UI Library, it perfectly aligns with our style guidelines. By customizing components, leveraging Emotion.sh for styling, and establishing a theme object for consistency, Hydrogen has revolutionized our UI development process. The result? Enhanced efficiency, improved cohesion, and an unparalleled user experience that leaves a lasting impression. Our commitment to Hydrogen has truly propelled us forward.

For a recent campaign, one of our customers wanted to use creatives with bright colours. While making these creatives on our web-based editor, it looked bright, but the generated output looked faded.

What the customer wanted vs what we generated

Our findings

When we analyzed the metadata of the output images, we realized that the colour profile was getting set to sRGB IEC61966-2.1.

A little intro on digital colours

Let’s try to understand colours and how they are displayed digitally.

Colour models - A colour model is a way to represent and describe colours. It defines how colours are created by combining different components. The most common colour models are RGB (Red, Green, Blue) and CMYK (Cyan, Magenta, Yellow, and Black).

Colour spaces define the number of colours available (sRGB, Display P3).

And colour profile instructs how to accurately display those colours on specific devices or applications.

The following animation shows the colours available in different colour spaces. Gray denotes the colour range most human eyes can see. The colours in the triangle are what is supported by digital screens. sRGB (the smaller triangle) used to be standard on the internet. But a lot has changed in the past few years, which becomes evident from this article. (Courtesy: wigglepixel.nl)

New learnings

1. We realized that since Chrome 94, display-p3 support was available for the canvas element.

opts = {colorSpace:'display-p3'};
const ctx = canvas.getContext('2d', opts);

2. Even though Chrome started support for display-p3 for users, an option to take screenshots with ‘display-p3’ colour profile via the Chrome Developer Protocol was missing at the time of writing this article:
   

How to retain the original colours?

The original colours used to generate the creation were in Display P3 (containing more expansive colours than sRGB). When we take the final screenshot using Puppeteer, Chrome takes the screenshots in sRGB. As Puppeteer took the screenshot in a headless Chrome instance (both in a lambda or a VM instance), no physical monitor was involved. A few of the approaches we took are the following:

1. Tweak the virtual display to render in a display-p3 colour profile.

2. Force chrome to render in display-p3 profile using experimental tags.

We started working on this issue around mid-April. We took time to break down different parts of our infrastructure, create standalone experiments, tweak, and test the results. Red lines show failed experiments, green shows successful experiments, and grey are pending experiments.

A significant shift in finding the solution happened when we took a step back and asked fundamental questions about the approaches taken.

Instead of tweaking the colour profiles of the virtual display, we thought of attaching the desired colour profile to the generated image. To quickly check our approach, we used the Assign Profile feature in the Preview tool on Mac to assign a Display P3 profile. The results were very close to the desired output.

An interim solution

The solution, for now, was to programmatically attach the Display-P3 colour profile after an image was generated.

With this knowledge, we were able to:

1. Bring support for a broader range of colours in our output.

2. Fix the fade-out issue during the import of PSD files.

3. Improve our colours in video output.

Drawbacks of the current approach

We still lose some details of the images even if we can retain the colours.

Resources

1. Colour space and FFMPEG

2. Colour Models

3. Colour spaces and colour profile

4. Expanding gamut of colours on the web

5. Google Chrome Developer Relations Portal

6. Oklab - A perceptual colour space for image processing

The author, after running and documenting the experiments.

Updates

• With the latest release of Chrome 114, we have access to more colours on the web:

• A deeper article about colours and colour-profiles on the web by Adam Argyle from the Chrome Dev Rel team.

  
  

When it comes to advertising, every detail matters. From captivating headlines to captivating images, advertisers constantly strive to create eye-catching ad creatives that leave a lasting impact on their audience. Among the many elements at play, two often overlooked yet crucial factors are the colors used and the sharpness of text within the creative.

“Crisp and sharp text is the anchor that holds an ad together. It ensures that the message cuts through the noise and makes a lasting impact on the viewer” - David Ogilvy, advertising legend and founder of Ogilvy & Mather.

Picture this: You're scrolling through your social media feed, and amidst a sea of blurry or pixelated ads, one stands out with crisp, razor-sharp text. Your eyes are instantly drawn to it, and within seconds, you've grasped its message. That's the power of sharp text – it grabs attention, communicates effectively, and ensures the viewer doesn't miss out on the intended message.

In this blog, we will discuss everyday issues we face when creating ad creatives from any web-based platform like Rocketium, Canva, or Photopea.

🌫️ Blurry Export: The Problem

Before jumping on to the issue, let’s discuss how to export work in Rocketium and how we generate images from HTML.

Any idea how to do it?
If your answer was to use Puppeteer and take a screenshot, Then we all are on the same page, if haven’t guessed it correctly. Let’s discuss how it works. At Rocketium we handle thousands of creatives that too with different file types, being a web-based platform we use Puppeteer to generate creatives. Under the hood, it opens a headless Chrome and takes a screenshot of the HTML output, and exports it in the required file format. So with this, we can generate anything on HTML and simply opens up a new headless Chrome tab and then take a screenshot there.

📸 When it comes to taking screenshots using Puppeteer, here are some major hurdles you may encounter:

1. Rendering Inconsistencies: One of the primary challenges with Puppeteer is dealing with rendering inconsistencies across different websites and web applications. Since Puppeteer relies on the Chromium browser, variations in CSS styles, JavaScript behavior, or dynamic content can result in screenshots that differ from what is expected. Ensuring consistent and accurate rendering across various scenarios can be a complex task.

2. Page Load Timing: Capturing screenshots at the precise moment when a page has finished loading its content is crucial for obtaining accurate representations of web pages. However, timing can be challenging, especially when dealing with pages that load dynamically or have asynchronous content. Ensuring that Puppeteer waits for all necessary resources to load before taking a screenshot requires careful handling and synchronization.

3. Performance and Resource Usage: Puppeteer operates by launching a headless Chromium browser, which consumes significant system resources. This can impact performance, especially when dealing with complex or resource-intensive web pages. Efficiently managing resources, optimizing code, and handling potential memory leaks are essential to maintain a smooth and reliable screenshot-capturing process.

4. Image Size and Resolution: The resolution and size of the captured screenshot can significantly impact its clarity. Puppeteer allows you to specify the viewport size and the dimensions of the screenshot, and using appropriate values is crucial for obtaining sharp and clear images. Inadequate resolution or incorrectly sized screenshots can result in blurry output.

5. Sharpness: The preview of any HTML Page depends totally upon the user screen, so while generating it totally depends upon which OS and machine you are using for your servers. This generates a lot of issues with the quality of images in the creative.

We have handled most of the issues and tackled them while setting up the infrastructure but recently we hit a bottleneck with blurry outputs.

While working with one of the customers, a customer reported that one of the product images is very blurry post output and when we dug deep, Puppeteer was at fault 🤯 .

We tried looking at the issue and the root cause was the screenshot. Till the screenshot the creative looked fine but when we take the screenshot the Image lose the sharpness. So we thought of doing some experiments to resolve this. Let’s look into what we tried.

🧪 Experiments

To resolve this issue, we spent 3 weeks trying different experiments to figure out if we can achieve some success.

🟡 - Improvement, 🔴 - No Improvement, 🟢 - Success

Some of the approaches we have tried -

1. 🟡 Scaling the preview to twice the size and resizing to the original

2. 🔴 Taking a manual screen grab on Mac.

3. 🔴 Taking a manual screen grab on Windows.

4. 🔴 Taking a screenshot using Puppeteer on Mac.

5. 🔴 Taking a screenshot using Chrome API.

6. 🔴 Taking a screenshot on a GPU machine.

7. 🔴 Taking a screenshot after adding CSS options to instruct the browser to render sharp.

8. 🔴 CSS - image-rendering: -webkit-optimize-contrast;

9. 🔴 CSS - transform: translateZ(1px) OR transform: translateZ(0);

10. 🔴 ‘srcset’ instead of src in <img> tag.

11. 🔴 Resize the product image to be the exact size it is supposed to be rendered first and then use the URL.

12. 🔴 Sharpen the image before and add that and then generate output.

13. 🔴 Run puppeteer on Mac M1 Air and Mac mini.

After trying all the experiments, we still couldn’t achieve the results. We were out of ideas. We had literally no idea what to do here, then Santosh ( VP at Engineering ) pitched to research around how Photoshop is able to achieve this whereas none of the web-based platforms are able to solve this problem.

🔬 The Research: First Step Towards Success

When we started researching how PhotoShop generates images, the first thing that we could sense was, PS generated everything client side, and that to pixel by pixel impainting. Using the user’s device power and capabilities it generated each creative by imprinting each pixel instead of a screenshot. Not only this then we came across a research paper and an article mentioning how sharpening is used post-export in various platforms and even in PS.

My first reaction to this article was, this was so obvious! If you have blurry output, instead of fixing the export, sharpen the export and TADA! As soon as this struck, we all had a ray of hope and an idea of what can be done here. We started reading articles about different techniques and algorithms that sharpen an Image, and believe me when I say there are 100s of algo, there are actually so many of them that we were so confused about which one to use and why? We read about each one of them ( Like not every but at least the top 10 ). Then we decided to look out for the options that we can integrate easily and solves for our use case!

Then we came across this Article, which explained how resizing works and how an image's brightness/blurry-ness happens. So we started digging more into it and then did some tests. There was so much to handle but finally, we hit the jackpot. We could see the difference

🔑Solution: The Final Approach

So now it was clear that sharpening will do the magic. So we finalized with ‘Image Magick’. We started looking at how Image Magic works and what algorithms it works on.

So we came across a command line tool from ImageMagick itself that will do this job for us. So we tried our hands on it and it had so many properties that we could play around with.
Let’s jump into some code and figure out how Image Magick works.

This is the ImageMagick command that does the magic (sharpening) for us.

magick sample.jpeg -unsharp 1.5x1+0.7+0.02 sharpened.jpeg

So here we can see there are multiple parameters that we need to pass on. Let’s look into it what all these parameters stand for.

So here unsharp is the main command that enables the sharpening algo. After that, we have the parameters. There are 4 values that determine how the logic will work and what it will change for colors and borders to make the image sharper.

Let’s Focus on these 4 parameters.

1. Radius (1.5) - The radius of the Gaussian, in pixels, not counting the center pixel (default 0).

2. Sigma (1) - The standard deviation of the Gaussian, in pixels (default 1.0).

3. Gain (0.7) - The fraction of the difference between the original and the blurred image that is added back into the original (default 1.0).

4. Threshold (0.02) - The threshold, as a fraction of QuantumRange, is needed to apply the difference amount (default 0.05).

So we can manipulate these values and can identify which works better for which use case. For Our use came up with these values.

🚀 Implementation

Our export works mostly on serverless so we had to install ImageMagick there and use it in code. So we installed it using a layer under GM and used it as a function where we can pass the values from the frontend so that if in the future we want to manipulate we can easily play with the values from the frontend itself.

So finally we were able to implement and sharpen images as per the requirements.

You can refer to the above function written in javascript :)

Everyone was happy and we were about to ship this as well, but it wasn’t this easy as well. While testing we came across some cases which just took our whole implementation to 2 steps back. We were sharpening the whole screenshot and exporting it but due to a major issue raised that even the text and the background image were sharpened we somehow ended up creating a jaggy effect for the text.

😱 The Climax

So after facing issues with the text, we decided to take another step and make this a bit more adaptable for our use case. So now after taking screenshots of the whole creation, we wrote some rules to figure out which all images need to be sharpened. We hide all other elements and just keep that image and take another screenshot of it and then sharpen it. Example -

And then we stitch this image back over the main creative. That should solve the issue, right? NO 😖

With this another issue comes that was what if there is an image above a text? This would overlap text. So now we sat back and finally came up with the final approach.

So now we take each element of the creative, hide other elements, and take a screenshot of it. So if we have 5 layers, we have 5 screenshots ready with us, with the rules we take this array and replace these images with sharpened images ( for all which are required ). After all, the sharpening is done, we send this to a function that takes all the images and stitches each of them over each other.

And TADA! We are finally done with our implementation and now you can easily get all the sharpened export even if Puppeteer doesn’t let you :)

In today's hyper-connected world, where business operations span across continents and remote work has become the new norm, the importance of secure and seamless connectivity cannot be overstated. For years, Virtual Private Networks (VPNs) have served as the go-to solution for organizations seeking to protect their data while enabling remote access. However, as technology evolves at a rapid pace, a new paradigm has emerged, promising to revolutionize the way we approach secure connectivity: Zero Trust Network Access (ZTNA).

In this technical blog, we will embark on an exciting journey that showcases how Rocketium made a remarkable transition from VPN to ZTNA, unshackling ourselves from the limitations of the past and embracing the full potential of this cutting-edge technology. We will explore the shortcomings of traditional VPNs, shed light on the core principles behind ZTNA, and delve into the practical implementation of this revolutionary approach.

The Need for Change: Challenges Faced with VPN Solution

In the quest for a more efficient and reliable network infrastructure, Rocketium decided to move away from the traditional VPN solution. While the VPN served its purpose for years, it had its fair share of challenges and issues that hindered our productivity. Here are the key problems we encountered:

• Sluggish Performance: The VPN often struggled to keep up with our growing demands, resulting in slow connection speeds and frustratingly long loading times. This greatly hampered our workflow efficiency and impacted employee productivity.

• Lack of Scalability: As our company expanded and our workforce grew, the VPN's limitations in accommodating increasing connections became evident. We needed a solution that could seamlessly scale with our evolving needs, which the VPN failed to provide.

• Geo-Restricted Websites: With business operations spanning multiple geographic locations, accessing geo-restricted websites became a hurdle. The VPN's inability to bypass such restrictions proved to be a significant inconvenience, hindering our international collaboration efforts.

• Unstable Connection: The VPN frequently suffered from an unstable connection, leading to unexpected disruptions during critical tasks. Whether it was dropped connections or intermittent service outages, these disruptions affected our team's ability to work efficiently and reliably.

• Call Quality Issues: VoIP calls were particularly challenging with the VPN. The unreliable connection resulted in distorted audio, call drops, and frustrating delays, making it difficult to communicate effectively with clients and colleagues.

  

• Notification Delays: The VPN's unreliable nature often caused delays in receiving crucial notifications and updates. This had a direct impact on our ability to respond promptly to time-sensitive matters, affecting our overall responsiveness as a company.

• Frequent Disconnections: One of the most frustrating issues with the VPN was the frequent disconnections experienced by our employees. These unexpected disruptions disrupted workflows, leading to frustration and decreased productivity.

• Speed vs. Reliability: The VPN's reliance on UDP for speed compromised its overall reliability while opting for TCP for reliability sacrificed speed. This trade-off presented a constant challenge, as neither option provided the ideal balance between speed and dependability.

Recognizing the limitations and challenges posed by the VPN solution, our company determined that a shift towards a Zero Trust Network Access (ZTNA) approach would provide us with a more secure, scalable, and reliable network infrastructure. By addressing these issues head-on, we are now better equipped to meet the demands of a modern, fast-paced business environment.

Introducing ZTNA: A Paradigm Shift in Network Access

Zero Trust Network Access (ZTNA) is a security framework that provides secure access to resources while adopting a Zero Trust Security approach or simply "never trust, always verify" approach. Unlike traditional VPN solutions, ZTNA focuses on verifying the identity and trustworthiness of users and devices before granting access to applications and resources.

ZTNA operates on the principle of granting access based on a combination of contextual factors such as user identity, device health, location, and application requirements. It eliminates the concept of a trusted network perimeter and instead treats every access request as potentially malicious or unauthorized.

Unlike VPNs that often provide unrestricted access once connected, ZTNA enforces granular, least-privileged access controls. It enables organizations to define fine-grained access policies, granting users access only to the specific resources they need for their roles or tasks. This approach enhances security by significantly reducing the attack surface and minimizing the risk of lateral movement within the network.

By shifting the focus from network-centric to user-centric security, ZTNA addresses the limitations of traditional VPN solutions and provides a more scalable, flexible, and secure network access framework for modern enterprises.

Working of ZTNA: A Deep Dive into Secure Access

Here's an overview of its working:

1. User Authentication: ZTNA begins by authenticating users using strong methods such as multi-factor authentication (MFA) to verify their identities before granting access.

2. Contextual Access Control: ZTNA leverages contextual factors such as user identity, device health, location, and application requirements to enforce granular access controls. Access is determined dynamically, allowing users to access only the resources they need based on the principle of least privilege.

3. Secure Broker: ZTNA employs a secure access broker, acting as an intermediary between users and the resources they want to access. This broker verifies user identities, checks device health, and enforces access policies to establish a secure connection.

4. Strong Encryption: ZTNA ensures secure communication between users and resources by utilizing robust encryption protocols such as Transport Layer Security (TLS). This encrypts data during transit, safeguarding it from unauthorized access.

5. Continuous Monitoring: ZTNA continuously monitors user activities, device health, and network behavior to detect and respond to potential threats. Real-time monitoring helps identify anomalous behavior and prevents security breaches

   

By understanding the inner workings of ZTNA, organizations can establish a secure and dynamic access model that protects against evolving threats while providing users with streamlined and efficient access to resources.

Benefits of ZTNA: Empowering Security, Simplicity, and Performance

After adopting Zero Trust Network Access (ZTNA), our company experienced a multitude of benefits that revolutionized our network infrastructure.

• Enhanced Security: ZTNA's principle of least privilege granted users access only to the specific resources they needed, reducing the attack surface and mitigating the risk of unauthorized access. This approach significantly enhanced our overall security posture.

• Simplified Network Management: ZTNA simplified network management by eliminating the need for complex VPN configurations and maintaining multiple access controls. Its centralized policy management streamlined administration and reduced complexity, resulting in more efficient network operations.

• Improved User Experience: Compared to the sluggish and unstable connection of our previous VPN, ZTNA offered an improved user experience and performance. With faster and more reliable access, employees could seamlessly connect to resources, access geo-restricted websites, and engage in stable voice calls. Delays in notifications and frequent disconnections became a thing of the past.

The adoption of ZTNA resolved the issues we faced with VPN, providing heightened security, simplified management, and an enhanced user experience. In the next section, we will explore the implementation process and the key considerations for a successful ZTNA deployment.

Migration Process: Transitioning from VPN to ZTNA

Transitioning Rocketium from a VPN solution to Zero Trust Network Access (ZTNA) involved a well-planned migration process. Here are the steps we took to ensure a smooth and successful transition:

1. Planning and Preparation: We conducted a thorough assessment of our existing network infrastructure, identifying dependencies and mapping out the resources and applications that needed to be accessed through ZTNA. This helped us create a migration plan, set goals, and allocate necessary resources.

2. Pilot Testing: Before implementing ZTNA across the entire organization, we conducted pilot testing with a small group of users. This allowed us to validate the functionality, compatibility, and performance of the ZTNA solution, and address any issues or challenges early on.

3. Communication and Training: We ensured effective communication with employees, explaining the benefits of ZTNA and providing training on the new access workflow. This helped ease the transition and ensured everyone understood the changes and how to use the new system effectively.

4. Rollout: During the rollout of ZTNA to employees, we implemented a strategy that involved maintaining VPN as a backup solution for a week, ensuring a seamless transition and addressing any potential issues that may arise.

By following a well-structured migration process, addressing challenges proactively, and ensuring effective communication and training, we successfully transitioned from VPN to ZTNA, empowering our organization with enhanced security, simplified management, and improved user experience.

Real-World Results: Tangible Benefits of ZTNA Implementation

The adoption of Zero Trust Network Access (ZTNA) yielded significant real-world results, showcasing the transformative impact it had on our organization. Here are specific examples and case studies that demonstrate the benefits of ZTNA:

1. Improved Internet Speed and Routing: With the Advent of ZTNA we have seen increased internet speed and better internet routing.

   

2. Cost Savings and better productivity: By adopting ZTNA, we realised cost savings across various aspects of our network infrastructure. Previously, our VPN incurred substantial costs, amounting to several hundred dollars, whereas the implementation of ZTNA proved significantly more cost-effective. Furthermore, the improved user experience and minimized downtime contributed to increased employee efficiency, resulting in measurable cost savings for the organization.

3. Positive Feedback: Employees and stakeholders expressed their satisfaction with the ZTNA solution. They appreciated the enhanced security measures, improved access experience, and the ability to seamlessly work across different locations and devices. Positive feedback highlighted increased job satisfaction and a more productive work environment.

These real-world results exemplify the measurable improvements in security, productivity, and cost savings achieved through the implementation of ZTNA. The positive feedback from employees and stakeholders further validates the effectiveness of this transformative networking approach. In the next section, we will delve into best practices for maintaining and optimizing a ZTNA environment to ensure long-term success.

Best Practices and Tips: Smooth Transition to ZTNA

For organisations considering a transition to Zero Trust Network Access (ZTNA), here are some valuable recommendations to ensure a successful implementation:

1. Selecting a ZTNA Solution:

   • Evaluate the scalability, performance, and security features of different ZTNA solutions to find the best fit for your organization.

   • Consider compatibility with existing systems and applications to ensure seamless integration.

   • Look for solutions that provide comprehensive access controls, granular policy management, and robust authentication mechanisms.

   • In our case, it was Cloudflare Access.

2. Training and Education:

   • Provide thorough training and educational resources to familiarize employees with the new access model and its benefits.

   • Highlight the importance of following security protocols and best practices to ensure a secure network environment.

   • Communicate the impact of ZTNA on productivity and user experience, addressing any concerns or misconceptions.

3. Collaborate with Stakeholders:

   • Engage key stakeholders, including IT teams, security professionals, and department heads, in the decision-making process.

   • Foster collaboration and open communication channels to gather feedback, address concerns, and ensure alignment throughout the transition.

By considering these best practices, organizations can navigate the transition to ZTNA smoothly, ensuring a secure and efficient network environment. Ongoing monitoring and adaptation, coupled with employee training, will help organizations stay ahead of evolving threats and maximize the benefits of ZTNA.

Conclusion: Embracing ZTNA for a Modern and Secure Future

In this blog, we explored the need for change from VPN to Zero Trust Network Access (ZTNA) and the challenges faced with VPN solutions. We then introduced ZTNA, highlighting its principles and how it differs from traditional VPNs.

We delved into the benefits of ZTNA, showcasing enhanced security through the principle of least privilege, simplified network management, and improved user experience compared to VPNs. Real-world results demonstrated measurable improvements in security, productivity, and cost savings.

The migration process involved careful planning, pilot testing, communication, and rollout.

To conclude, the adoption of ZTNA empowered our organization with heightened security, simplified network management, and improved productivity. We encourage readers to consider ZTNA as a modern and secure alternative to VPNs, offering granular access controls, enhanced user experience, and adaptability to evolving threats. Embrace ZTNA for a future-ready network infrastructure that prioritizes security, scalability, and productivity.

To gain deeper insights into our culture and processes, please visit culture.rocketium.com. At Rocketium, we cultivate an environment of collaboration and openness, where we work alongside highly talented and motivated individuals. If you're enthusiastic about joining our journey and becoming part of this thrilling adventure, don't hesitate to contact us at careers@rocketium.com.

Welcome to the realm of compliance, where rules and technology harmoniously intersect to uphold the integrity of brands. Compliance plays a vital role in maintaining a consistent and authentic brand experience, safeguarding consumer trust and loyalty.

In this journey, we will explore the importance of compliance and how we created a fusion of Machine Learning, OCR, NLP models, and our Mathematics expertise to figure out the problems.

What is a Brand?

A brand refers to the overall perception and image of a company, product, or service in the minds of consumers. It encompasses various elements such as the company's name, logo, design, messaging, values, and customer experiences. A brand represents the unique identity and reputation of a business, distinguishing it from competitors in the market.

Compliance, in the context of branding, refers to adhering to specific guidelines and standards set by a company to ensure consistency and integrity in the representation of its brand across different channels and touch-points. These guidelines typically cover aspects such as logo usage, typography, color palette, imagery, voice and tone, and other visual and verbal elements.

A small dig into what happened with Volkswagen

VW marketed their diesel vehicles as eco-friendly and compliant with strict emission standards. However, it was later discovered that the software in their cars could detect when the vehicle was undergoing an emissions test and adjust the engine's performance to meet the requirements. In regular driving conditions, the vehicles emitted nitrogen oxide (NOx) pollutants well above the acceptable limits, contributing to air pollution and environmental harm.

The scandal had significant consequences for Volkswagen, including substantial financial penalties, a damaged reputation, and legal ramifications. The brand's integrity and trust were severely impacted, as customers and stakeholders felt deceived and betrayed by VW's actions.

This is how serious things can happen in cases of compliance breach.

How is it done presently?

Currently, compliance validation is primarily performed by experienced professionals with in-depth knowledge of the specific guidelines and regulations governing brand compliance. These experts manually review creative content, assessing various elements such as typography, colors, imagery, branding consistency, legal requirements, and other relevant factors. They compare the content against predefined compliance guidelines and identify any deviations or violations.

Market Research

Open any e-commerce app or visit any well-known brand's website and observe the consistency across their images.

After observing the same we analysed some trends :

1. Logos are usually placed at corners or centre of the creative.

2. T&C is very less font size and placed usually at a bottom corner.

3. Overall 40-50% white-space to get better visibility.

4. Total number of character limitations.

5. In some ad-spaces we also observed some UI portion being acquired, so in that place nothing should be existing.

6. Use of certain colors.

7. No mention of any designations etc.

8. Gender and diversity neutral policies.

9. No support of alcohol.

The fusion of ML, OCR, and NLP?

So after analysing the problem we realised the need of the following and broke the problem into following pieces :

1. Segregating different pieces of image, heading, sub-heading, call-to-action, terms-and-condition, logos.

2. Counting text, doing analysis on it etc.

3. Asking questions to the image or text inside it on certain things like designation mentions, title or sentence case etc.

To analyse the above problem we did the following :

1. Using OCR technology to figure out all the text inside the image.

2. Using machine learning to train from each account creatives and identifying different labels.

3. Using NLP models to identify some things like designation, religion, alcohol bottles, etc.

   

Problems in combining three different technologies

While combining three different tech’s the major problem was identifying their super powers and playing to strengths. And all these will have errors and combining three different techs will cause increase of error and multiplication of those.

Super powers elaborated :

1. OCR is great in identifying all the text inside the creative. But won’t be able to classification.

2. ML or image classification will be good at identifying major pieces but won’t know what is inside it.

3. NLP models will help us in analysing some important rules. But not much helpful in classification as just text won’t give the whole perception of a creative.

Mixing magic

So the process followed :

1. Gather all the information from OCR.

2. Take out the data from our image classification model.

3. Try intersecting the image classification data and intersecting the data from OCR.

4. Tweak a couple of values around overlap percent and all.

5. Iterate, test and repeat.

Rule engine

After separating and mixing all the details the final thing was to put all this behind an engine for us to evaluate. This was using a JSON rule engine library, and extending it to run the things we need.

We added a workflow of two types of rule parts in general :

• filter : Filter elements based on any parameter or rule.

• aggregate : Aggregate the array we got from filter or already existing and get sum, or any other operations

A small example was having a requirement where elements do not intersect with a Person’s image. So for this we wrote a small piece to detect if a single position is intersecting with the other. Sample code below, with the rule engine JSON attached.

Cost analysis

After all this is done some of these cost should always be a factor in any project to make it business viable :

1. Machine learning and these NLP models are slightly on the costlier side but there are ways to reduce some cost.

2. Try using batch predictions instead of an online prediction, although that also depends on scale and user experience need to be provided.

3. Building own models and deploying the same behind kubernetees.

Future scope and story

Rocketium aims to become the ultimate destination for comprehensive creative validation services. As a one-stop solution, we empower brands to effortlessly validate their creatives for advertising on any platform. Our platform eliminates the need for constant back-and-forth between teams by efficiently handling minor ad modifications. By entrusting us with the process, designers can save valuable time and focus on their core tasks, confident that their creatives will meet the necessary requirements seamlessly. With Rocketium, brands can streamline their validation workflows and ensure a hassle-free experience from start to finish.

“Can we search for the photoshoot image from last year where a female model was wearing an orange dress, we remember we uploaded it on Rocketium, but don’t remember the exact name of the image.”

Normally users were having a strong nomenclature to their images and for years they have been searching based on certain tags and names, sorting them by date and they were good to go. There was some manual effort involved, but no method existed for figuring out what was in that image. And that’s exactly what they wanted - If we can search their images by understanding the content of the image as well?

When I learnt about this problem surfacing, I was leading the development of our project workflow management tools which allow creative teams to streamline their project and asset management inside Rocketium, so that their designers have everything in one place. This search problem seemed interesting and I knew it could be done, I just had not thought how, but I had the zeal which runs in every engineer here - “We’ll figure out the tech to make this work” 💪

Our current architecture ⚙️

At Rocketium we use MongoDB as our primary database to store general user data like project data, asset metadata, etc and our assets reside in specific AWS S3 buckets for different users considering data privacy. Pretty standard stuff I’d say. I personally believe great engineers became great by leveraging the resources they had to the best use and not by reinventing the wheel every time they had a new problem. That’s what I had to do - enhance the existing infra with image content identification and search capabilities.

Now with this architecture, I had two ways to go about it → go make a presentation and present to the team that I can do it and convince them of whatever solution I proposed and then go build it OR I can just quickly build a prototype for them to have a look and feel of what I want to build in the product - I always chose the latter.

At Rocketium Engineering, we follow a culture which dictates that “Code wins over arguments”, so if you want the team to be convinced of your solution, the simpler way is to build out a proof of concept which will help you understand the problem better, and also the team to understand your solution better. The faster you can prototype, the faster you can get to the implementing part. That speed and agility are what I’ve learnt here, and I had just one week to research and prototype this.

Understanding the image content 🧠

The first part of solving the search problem was to identify what is in the image? One naive solution is to just implement any image recognition service like Google Vision and ask it to tag the images directly, but where’s the fun in that?

Any solution is worthless without adding the context of its application. We need to understand who is gonna use this feature and build it keeping that persona in mind, while keeping it flexible enough to cater to changing needs.

And for us, the context was that we deal with mostly advertisement campaign creatives which have a few elements - the model image, the product, a background, and the text (heading, subheading, offer text, and CTA text). All of these assets are separately stored in our media library which can be combined to build these ad creatives inside our campaign editor.

The power of image recognition in this context would come from actually being able to bifurcate the tags as well into categories. Remember what the customer asked us - “Can we search for the photoshoot image from last year where a female model was wearing an orange dress”. What if we could identify the products, colours, humans, and other things in the image and place them in separate categories so that users can search across these verticals! 🧐

We were already using Google Vision to get the tags for every uploaded image, but they weren’t that accurate, let alone categorisation. So I fed them to Chat GPT API and asked it to classify them into categories. It was smart enough, but not so much.

Next, I thought of training my own ML model for this and I figured out that Amazon Rekognition has Custom Labels which we can use to label our images with a custom dataset of labelled images.

That worked decently well where the training set was just 20 images and the test set was 10 images. The model had an F1 score (or accuracy) of ~60% but there were two challenges with it:

1. We need a large variety of images to train the model of different types of products and human images.

2. We need to label the images manually to train on them, which will become a pain real soon because of the volume of images we handle.

Next, I thought, why not use the default pre-trained models from Amazon Recognition, this led to a realisation. Since Amazon and Rocketium kind of work in the same domain of e-commerce, where Amazon sells these items, Rocketium helps create ads which sell these items. We both need product and model image recognition for classification. 🤝

Amazon Rekognition works pretty well when it comes to identifying the products and humans, plus they have also classified their data into categories which an e-commerce site could benefit from. They also have the capability to identify dominant colours for the foreground and the background.

Tagging images with Amazon Rekognition 🏷️

Every image document stored in MongoDB has certain properties like name, URL etc, and I added a “customFields” key to it which would be an object that can store these key-value pairs

Amazon Rekognition gives an elaborate response when it comes to image labelling, now it was up to us on how we can utilise this data for our use case. Rekognition gives us a category of the label which I planned to map to the “customFields.product” key of the image metadata, and only selected the top 10 labels which were predicted with more than 75% confidence.

For the colours, I was taking “Foreground.DominantColours” returned from Rekognition and returning them as one of the 12 natural colours. The tags which were returned but did not fit the categories we mentioned earlier were returned as just “tags” for safekeeping.

The thing to note is that all of these tags were just suggestions for the Humans, generated by AI. No way that we were forcing these labels on the images. Once the labels are received, anyone doing the quality check (QC) could modify them, but they get a great starting point.

Searching through these tags 🔎

Now using the powers given to us by Amazon Rekognition and applying some JS magic, we were able to tag every image with a “product” list, and “colour” list, and assign some other properties to it as well through “tags”. But this was just half the story - the next one is even trickier. Enabling users to search images by just writing in a natural language like → “Orange dress woman” and get images of women wearing orange dresses!

There are two ways which every engineer on some research would find when it comes to searching for documents in MongoDB.

1. Sync your data with Elastic Search (with its mongo-connector) and use the full-blown powers of Elastic Search to search your data.

2. If you have a self-hosted MongoDB, then create a Text Index and then use the $text operator in the query to search through documents.

But there are two potential challenges with this as well, which I found during my research:

1. There is always a maintenance cost associated with introducing a technology and I wanted to keep it as minimal and light as possible so that it just works, without much maintenance effort.

2. The $text Search in Mongo is a full-word search, so if you plan to search “Chi” in a document which has the text, “The Chinese built the great wall of China”, it will not be returned in the result. But we needed a partial search as well if you also need autocomplete

Then? How do I achieve a powerful search functionality with autocomplete in my MongoDB collection?

Enter MongoDB Atlas Search ✨

Around 2020, MongoDB folks launched Atlas Search which comes with their cloud-hosted solution MongoDB Atlas. At Rocketium, we moved from a self-hosted solution to MongoDB Atlas, towards the end of 2022 with a humongous effort from and team. This enabled us to take advantage of Altas Search for this use case.

Apparently MongoDB Atlas and ElasticSearch both use the same technology under the hood which is - Apache Lucene and use similar concepts for building a search index. MongoDB Atlas seemed to be a great choice for us to build a search because -

1. There was minute maintenance overhead since we were already using and managing MongoDB Atlas for our db instances.

2. It did not require any separate infrastructure setup and we can just build queries for searching where our data already resided. No need for external syncing.

3. MongoDB Atlas automatically syncs the data and updates the search index for the newly added items. Set it and forget it.

The next challenge was to learn how to build and deploy a search index because apparently, that is not as simple as setting up Altas Search 😅

Building a search index for assets 📑

On the MongoDB Altas platform, you can create a search index which starts with a default value of:-

{
  "mappings": {
    "dynamic": true
  }
}

This will basically index everything in your documents and you can search on it, now while this may sound lucrative to do, it comes with the cost of heavy search indexes, and then speed.

Hence we disabled the dynamic mappings to give our own mappings, something like this

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "customFields": {
        "fields": {
          "colors": {
            "type": "string"
          },
          "products": {
            "type": "string"
          },
          "tags": {
            "type": "string"
          }
        },
        "type": "document"
      },
      "name": {
        "type": "string",
        "analyzer": "lucene.standard",
        "multi": {
          "keywordAnalyzer": {
            "type": "string",
            "analyzer": "lucene.keyword"
          }
        }
      }
    }
  }
}

The above search index is explained as follows —

• mappings: This section defines the schema for the search index.

  • dynamic: The value false means that the index won't automatically index new fields that get added to the documents in the future. If dynamic was true, then any new field added to a document would automatically be included in the search index.

  • fields: This section describes the fields to be indexed.

    • customFields: This is a document field (nested field) containing other fields.

      • fields:

        • colors, products, tags: These fields are set to be indexed as strings. This means that MongoDB will create a text index on these fields to allow text search.

    • name: This field is indexed as a string.

      • string: This allows text search on the name field. This field is further analyzed by two analyzers.

        • lucene.standard: This analyzer tokenizes text into terms (words), changes the tokens to lowercase and removes common English words.

        • lucene.keyword: This analyzer treats the whole text field as a single term. This is useful for exact match searches.

Now depending on the use case, this search index could be different to support different applications.

Once this is saved, MongoDB will start creating a search index which can later be utilised in our code, and these fields could be searched with the $search operator.

This example query will match name and products from the defaultIndex and return the results which match. We use this query as one step in the MongoDB aggregation pipeline.

{
      $search: {
        index: "defaultIndex", // index name
        compound: {
          should: [
            {
              text: {
                query: searchString as string,
                path: "name",
                fuzzy: {
                  maxEdits: 2,
                },
              },
            },
            {
              text: {
                query: searchString as string,
                path: "customFields.products",
                fuzzy: {
                  maxEdits: 2,
                },
              },
            },
          ],
        },
        highlight: {
          path: [
            "name",
            "customFields.products",
          ],
        },
      },
    },

Bonus 💡: This search index also allows us some typo tolerance by specifying fuzzy property with maxEdits, where we can specify how many characters can be misspelt in a word.

And the highlight field would return an object which can help you identify matched words in the search results.

And what about partial searching?

The way I designed this search index was to match exact words, but what about partial search as well? Well, that is covered under an Autocomplete index.

An autocomplete index helps in tokenising the text to search for partial parts of the text as well. It requires specifying “autocomplete” properties which help us fine-tune how we wish to build the index. Suppose we wish to just add autocomplete for the products field, the search index would look something like this —

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "customFields": {
        "type": "document",
        "fields": {
          "products": [
            {
              "type": "string"
            },
            {
              "foldDiacritics": false,
              "maxGrams": 8,
              "minGrams": 2,
              "tokenization": "edgeGram",
              "type": "autocomplete"
            }
          ]
        }
      }
    }
  }
}

Where the rest of the things are the same, but the “type: autocomplete” object properties are explained as follows —

autocomplete: This type is specialized for providing autocomplete functionality. This means, as users start typing into the search field, suggestions that match their input will be displayed, enhancing the search user interface.

• foldDiacritics: When set to false, this means the search will distinguish between characters with diacritics (like e, é, è, etc.) and their simple counterparts (like e).

• maxGrams and minGrams: These properties control the size of the text fragments (grams) that are indexed. In this case, the text will be broken into fragments that are at least 2 characters long (minGrams) and at most 8 characters (maxGrams) long. This affects how the autocomplete suggestions are generated.

• tokenization: It's set as edgeGram, which means the system will index the beginning fragments of the words. This is particularly useful for autocomplete functionality, where you want to match input from the start of a word.

This gives us two separate search indexes for exact match and partial match. We can keep both indexes combined in a single one as well, but that would just take more time to build since autocomplete indexes generally take longer to build.

Wrapping up

While MongoDB Atlas Search offers powerful full-text search capabilities, it comes with potential challenges related to cost, speed, and scalability, especially at a larger scale. As with any technology solution, it's essential to balance the requirements of functionality, cost-effectiveness, and performance.

But building an effective search within our media library was one of the most challenging and interesting problem statements I’ve worked on in the past year, and wrapping up the research and prototyping in a week was thrilling!

And btw this whole code is open-sourced if you wish to have a look - https://github.com/viveknigam3003/cross-search

Work at Rocketium

At Rocketium, you'll tackle complex challenges of scale, drive innovation, and shape the future of enterprise solutions. If you're passionate about problem-solving and eager to influence how enterprises manage data at scale, have a look at our careers page or drop a mail at careers@rocketium.com

More about the amazing culture we follow at Rocketium, have a look at culture.rocketium.com

Introduction

Embracing the rapidly evolving field of artificial intelligence (AI), generative AI has proven to be a transformative force. Its potential goes beyond mere trends, as integrating AI capabilities into your products has become essential. However, with generative AI being relatively new and full of untapped possibilities, it can be challenging to determine where to begin. In this blog, we will provide insights on how to approach generative AI, offering practical examples from Rocketium's implementation plans. These real-world scenarios will serve as inspiration for incorporating generative AI into your own product. It is assumed that you have a basic understanding of generative AI concepts and terminology.

Choosing Your High-Level Goals

Start by assessing which of these overarching objectives align with your business, enabling you to focus on specific use cases where Generative AI can be applied effectively.

1. Creativity and Innovation: Generative AI can break the mold, fostering the generation of novel ideas, designs, and content.

2. Personalization and Customization: Offering customized services and products enhances user satisfaction. Generative AI can curate experiences tailored to individual users, understanding user preferences on a granular level.

3. Efficiency and Automation: Generative AI can automate various tasks, reducing manual effort, and increasing efficiency across several domains such as content creation, product design, and manufacturing.

4. Enhanced User Experiences: Generative AI can create immersive, interactive, and engaging content to enrich user experiences.

5. Problem Solving and Optimization: Generative AI can aid in resource allocation, decision-making, and process optimization by analyzing extensive data and generating potential solutions.

6. Data Generation and Augmentation: Generative AI can generate synthetic data to supplement existing datasets, improving machine learning models' performance.

Exploring the Domains of Generative AI

After determining your goals, you can explore generative AI within the following contexts, which are particularly relevant in the near future. In the next section, we will delve into each area and discuss our thoughts and strategies for implementation.

•  Text -> Text

  • Text-to-text transformations involve AI models like GPT-4 that predict the next word in a sequence based on the preceding words, enabling generation of contextually related sentences and paragraphs. Examples include machine translation and text summarization.

• Text -> Image

  • Text-to-image generation involves creating visual representations from textual descriptions using models like GANs. A well-known example is DALL-E, which generates images from textual prompts. Applications include art, design, and virtual reality.  

• Image -> Text

  • Image-to-text conversions employ transformer models for generating text. These AI models craft textual descriptions for images, significantly enhancing the capability to semantically search for an image via text. Refer https://minigpt-4.github.io/

• Image -> Image

  • Image-to-image transformations modify an input image to generate a new output image using Generative Adversarial Networks (GANs). Examples include generating variations of an image, upscaling an image, removing part of an image, and extending an image,… Applications are wide, ranging from art and entertainment to autonomous driving and medical imaging.

Once you have chosen a specific area to focus on, you can start by exploring existing stock solutions available in that domain. It is recommended to begin with these stock solutions as a starting point. However, the true value of generative AI is realized when you are able to generate brand-compliant content that aligns with your company's language, tone, and voice. To achieve this, you will need to delve into the process of fine-tuning the AI models, ensuring they are customized to meet your specific requirements.

Finetuning, Embeddings, or Both?

To achieve brand compliance and generate content that aligns with your company's language and tone, fine-tuning and embedding techniques are crucial. Based on experiments conducted, both approaches have yielded similar results. However, fine-tuning is recommended when you have specific data sets that differ from the general set while embedding and feeding context to a language model has produced the best results for most use cases. There are a lot of articles written on this topic. So we will elaborate more on this. Below are some great reference materials that can help you with fine-tuning and embedding.

• Create embeddings

• How to finetune chatgpt

• Prompt engineering

Implementing Generative AI in Rocketium's Creative Operations

Here are a series of initiatives we have slated to incorporate Generative AI. More importantly, these should serve as an example to draw from and inspiration to incorporate into your own product development.

1. Brand-Compliant Creative Copy Generation: Advanced language models (LLMs) like GPT-4 can be trained on a company's past ad copies, which can be vectorized and stored in databases such as Elasticsearch. When a new copy is needed, a query can be run to fetch the most similar past copy vectors, and this copy can be used as a context for the model to generate a new, brand-compliant copy. The success of this process highly depends on precise prompt engineering, ensuring you communicate effectively with the AI.

2. Design Compliance: Using Generative Adversarial Networks (GANs), AI models can be trained on past ad designs. GANs are designed to create new data with the same statistics as the training set. Hence, they can generate new, brand-compliant designs that are similar to previous advertisements but carry a freshness in their aesthetics.

3. Media Library Insights: Generative AI can be used to extract metadata from media assets (using CNNs for images and LLMs for any associated text), which can then be vectorized and stored for semantic search. This is a significant improvement over conventional methods which rely only on text metadata and exact matching algorithms.

4. Image Utility Tools: AI offers a variety of tools for image manipulation, such as super-resolution (upscaling an image without losing detail, often achieved through models like SRGAN), background extension (using texture synthesis or inpainting techniques), and object removal (using convolutional neural networks that can be trained to recognize and erase certain image parts).

5. Improved Project Briefs: AI can use Named Entity Recognition (NER) to identify if crucial elements (like audience, objectives, channel, tone) are missing in project briefs. Additionally, LLMs like GPT-4 can be used to rewrite the brief in a format preferred by the company, maintaining brand consistency and clarity of communication.

6. Mood Board Generation: Models like GANs, trained on various aesthetic styles and visual elements, can generate images that fit a specific mood or theme. These AI-generated images can be dynamically added to a mood board, aiding in the creative brainstorming process for ad campaigns.

7. Stock Content Generation: AI can also generate brand-consistent stock content using GANs. By training on a diverse range of stock images, AI can generate new, unique variations of images that can be used in advertising material.

8. Creative Compliance: Generative AI can help improve creative compliance by identifying non-compliant elements. Using a combination of Computer Vision and Natural Language Processing (NLP) techniques, creative content can be deconstructed into its constituting elements (like text, image, color schemes, logos) and each element can be checked for compliance with brand guidelines.

9. Creative Analytics: By deconstructing ads into text and image components using NLP and Computer Vision techniques, AI can help analyze the impact of individual creative elements on ad performance. For example, attention-based models could identify parts of the ad copy that most effectively engage users. The results can be mapped onto standard marketing frameworks like AIDA (Attention, Interest, Desire, Action) or RFM (Recency, Frequency, Monetary) to provide deeper insights into ad performance. Moreover, these insights can be presented in a more digestible format by generating text-based summaries using LLMs, easing the interpretation of complex data.

Considerations for Generative AI Implementation

• Cost - Utilizing pre-existing platforms such as Google’s AutoML, AWS Bedrock or ChatGPT can significantly inflate costs, especially considering that ChatGPT 4 is nearly ten times costlier than ChatGPT 3, necessitating careful deliberation on the appropriate use of GPT3 vs. GPT4. Training and hosting your own Machine Learning (ML) models may provide substantial savings at scale, albeit with an increased demand for resources and expertise for system maintenance. The recommendation is to explore efficiency strategies for these hosted solutions. One example includes a proof of concept, which involved labeling various parts of creatives at a cost of $1500 for 500 images, with additional training costs as a one-time $100 fee. Querying costs present two differing impacts on user experience: Option 1 ($1500 per month), which provides a dedicated, always-online instance for immediate compliance checks, and Option 2 ($150 per month), which necessitates batching requests in a CSV format and submitting them, yielding results typically within 1-2 hours. Long story short, the cost can vary widely and you need to explore what is the right option for you.

• Time - Although pre-existing solutions can expedite your AI implementation, training and hosting your own ML model can be a time-consuming process. Developing proficiency in maintaining your own solution also requires a significant investment of time.

• Expertise - While having an internal team proficient in AI/ML is advantageous, the rapid advancements in this field have made AI accessible to a large number of engineers. Hence, there should be no hesitation in exploring this area.

Challenges and Future Scope

The challenges in implementing generative AI lie in training models to generate brand-compliant design ideas and adapting ads from one size to another without losing their essence.

As we venture further into the world of generative AI, we anticipate a shift. Once the initial euphoria around generative AI subsides, use cases that serve the invariables - efficiency, productivity, and creativity - will persist, and the rest will fade away.

Embrace generative AI in your business. It is not just an option, but a necessary leap into the future. Happy coding, and until next time, may the AI be with you!

For more insights into our culture and processes, visit culture.rocketium.com. We foster a collaborative and open environment, working alongside talented and driven individuals. If you are eager to join our journey and be a part of this exciting adventure, please reach out to us at careers@rocketium.com.

At Rocketium, we believe that a healthy work culture is the backbone of a successful team. This belief has led us to implement a system that draws from the wisdom in the book Radical Candor by Kim Scott. This system, designed to foster both praise and criticism, was initially introduced by Dan Wood, CTO of a start-up in the 1990s. It revolves around two main components, or rather, two stuffed animals: a whale and a monkey.

The "Killer Whale" of Achievement 🦈

Every Monday at Rocketium, our engineering team members have the opportunity to nominate themselves or a teammate for the "Killer Whale" award, which is conferred for a week. This award encourages team members to openly discuss the extraordinary work they've completed during the previous week.

As opposed to an employee-of-the-month style approach, the Killer Whale format ensures that those who do outstanding work get recognition promptly, at the beginning of the following week. If there are multiple nominees, the award's recipient is determined by a team vote, further emphasizing our focus on collaborative decision-making.

The Killer Whale mechanism motivates the team to strive for excellence and sets the tone for the week ahead. It promotes an environment that encourages individuals to take the initiative, innovate, and share their accomplishments with the team.

Embracing Mistakes with "Whoops the Monkey" 🐒

The second part of our system involves another plush toy and an equally important lesson: recognizing our mistakes. Team members can nominate themselves for the "Whoops the Monkey" award, designed to encourage open discussions about errors made in the previous week.

When someone owns up to a mistake they've made, they receive the monkey, symbolic of the automatic forgiveness that comes with it. This process creates an open, transparent atmosphere where it's okay to mess up, as long as you learn from it. The monkey is also a gentle reminder to others to avoid repeating the same mistake.

Bringing it to Life

To enhance visibility and make the process more interactive, we display photos of the winners with each toy on a wall in our office. Additionally, winners change their Slack profile pictures to the respective toys for a week.

While we've only been practicing this system for a while, it generates excitement every Monday morning as team members await the nomination process.

In Summary

The Killer Whale and Whoops the Monkey system at Rocketium aims to create a culture that values both achievements and learning from mistakes. While it's too early to measure its long-term impact, the system has invigorated our team dynamics and turned Monday mornings into a celebration of accomplishments and growth. Through this, we're fostering a work environment where praise, constructive criticism, and learning go hand-in-hand.

You can read more about our culture and our processes at culture.rocketium.com. We work with highly motivated and smart people in an open and collaborative environment. If you would like to be on this rocket ship, do write to careers@rocketium.com.

Goal-setting is one of the most important steps in building high-performing teams. But building an effective set of goals can be challenging and a time-consuming task. In this blog post, I will share a framework we use at Rocketium to define our Engineering goals.

How we set the goals

Start with company objectives

Having a clear understanding of the company's mission and objectives is the first step in goal-setting. Team and individual goals need to align with both.

Organise the goals around 3P framework

Taking inspiration from the People, Process, Technology framework, we have created our own People, Process, and Product framework. Every goal is mapped to a theme which, in turn, is mapped to one of the verticals of the 3 Ps. Below are the themes we defined under each category.

• People - Career objective, Collaboration, Visibility, Hiring, Mentoring

• Process - Communication, Feedback, Ownership, Collaboration, Validation

• Product - Development, Quality, Shipping, Product mindset, Support

Make goal-setting a collaborative exercise

Every team needs to work in unison for an organisation to run like a well-oiled machine. The entire Engineering team at Rocketium participates in goal-setting, seeking feedback from all other teams to ensure each team’s expectations are met.

Sharpen each goal using SMART framework

SMART stands for Specific, Measurable, Achievable, Relevant, and Time-bound. Defining these parameters for every goal ensures they are achievable within a specific timeframe. We have to admit it was hard to apply all parts of this framework for every goal but we did the best we could.

Engineering goals

The 3P framework helped us come up with strong initiatives. However, it is not the best way to organise goals as there can be overlaps across themes. Once we had all the goals, we regrouped them under common themes that emerged.

Below are all the goals we defined for our Engineering team across seniority levels.

Career objective

• Promotions and Rating - Work with every team member to define their career goals - promotion and performance ratings. This helps in setting the right expectation to achieve the goals upfront and provides periodic feedback to help the individual achieve those goals.

Visibility

• Increase your presence in the office. Share kudos and actively participate in relevant online and in-person discussions.

• Participate actively in meetings. Share at least one opinion, question, or feedback in every meeting.

• Create excitement about your work by sharing regular updates. Do a demo with relevant teams for early feedback.

• Write about your work on social media and your own or the team’s blog.

Collaboration

• Meet with the Product and Design team to discuss features and priorities. Work with them to ensure the feature list and design are ready before each Sprint.

• Work closely with the Customer Success team to understand product gaps. Work closely with the design team to launch high-quality features with the right user experience.

• Create a collaborative environment within the team to increase the quality of what we are working on. Brainstorm with the rest of the team and seek feedback before starting a major feature.

• Conduct one session a month showcasing product capabilities. Do this in person but also share a recording for future use. Capture this in our internal wiki.

• Build connections outside of the dev team by having lunch or coffee with at least 1 Customer Success / Design / Sales team member every month.

• Seek feedback from at least 1 Customer Success / Design / Engineering / Product team member you closely work with. Get positive feedback from 2 people on technical, communication, and written skills.

• Provide timely feedback to peers, managers, and external team members. Make notes during meetings and 1:1s. Take action before the next meeting.

Development

• Reduce complexity and promote reuse. Review the architecture of peer team members and provide feedback. Find elegant solutions to complex problems

• Document technical architecture, OKRs, manual test plan, and roll-out plan. Seek feedback from other team members for features before development starts.

• Identify ways to reuse code and create components so we add only what is needed.

• Conduct stringent code reviews. Reject those that do not follow coding best practices. Ensure all corner cases are covered. All new features have enough comments and logs to help debug the feature.

Technical leadership

• Spend 10% of your time inspiring and guiding a team member to complete their projects with outstanding results.

• Meet 3 new team members every week to take feedback on what you are working on, educate them on the feature area, and inspire them.

Shipping

• Ensure there is no slippage in planned tasks in a Sprint. The tasks should go live the week that the Sprint ends.

• Get better at estimating timelines and escalate when things are getting delayed. Clearly communicate timelines and blockers.

• Ensure the Design and Product team reviews the feature before we ship. Build the necessary environments for the teams to review on time.

Quality

• Ensure features are shipped with the highest quality and zero bugs.

• Manual testing and peer testing followed by unit and automated tests are a must for a feature release. Ensure features and adjacent areas are tested thoroughly. The features should be tested in an environment that closely mimics the customer’s environment.

• Track metrics on product quality.

• Ask the right questions and assess the priority of on-call issues. Relevant teams should be notified regularly and the root cause is shared and documented during closure.

• Ensure coding best practices are followed. UI must be aligned with brand guidelines.

Personal Goals

• Identify at least 1 broken process and improve them. Introduce at least 1 new process to solve existing challenges.

• Connect with at least 2 high-achievers on LinkedIn, learn from their journey, and share their success stories with the team.

• Read 6 books by the end of the year.

• Host 3 senior engineering leaders at our office for an architecture review or brainstorming.

Hiring and mentoring

• Be part of new hire interviews.

• Successfully mentor one peer or intern. Motivate others in the team. Be someone who the team looks up to.

• Constantly focuses on learning outside of daily work.

A rating (Above Expectations)

• Be inquisitive about our customer’s nuanced use cases. Review specific campaigns our customers are running

• Work hard on written and oral communication. This will help in internal discussions and customer calls.

• Build a brand on social media. Complete one technical book. Take one technical class. Attend one technical conference.

• Listen to at least one customer demo a week.

• Adopt a product-minded engineering mindset. Understand why decisions are made, how people use the product, and be involved in making product decisions.

• Building analytics dashboards for reviewing feature adoption. Spot patterns of poor or excellent usability from metrics. Interact regularly with customers to understand how our products can be improved.

• Own features from development to adoption.

• Constantly review, share and implement the latest and greatest findings in the engineering space - coding best practices, frameworks, libraries, ….

• Gain expertise in architecting large systems and products.

• Proactively identify the potential impact of bugs on all the customers using the feature, notify relevant teams, and build a plan to address it. Adopt the learnings in future work.

Conclusion

Effective goal-setting is a continuous process and crucial for building high-performing teams. Doing this collaboratively helps align each team to achieve the company goals effectively. Sharing the goals openly is a social commitment that inspires the team even more to achieve the result.

I hope this blog post helps you in building out your goals and/or your team’s goals. You can read more about our culture and our processes at culture.rocketium.com.

We work with highly motivated and smart people in an open and collaborative environment. If you would like to be on this rocket ship, do write to careers@rocketium.com.

In part 1, we covered the challenges in personalizing visuals at scale and how teams can use Smart Links to solve these challenges. Let us switch gears and dive into the technical details of Smart Links. We will cover:-

1. Search Algorithm

2. System Design

Search Algorithm

Problem Statement: Find the k most relevant variants from the n variants in a project

Input: n creative URLs, each with a set of m key-value pairs of metadata

Expected Output: k most relevant creative URLs

Algorithm

We built a modified version of K-nearest neighbors to solve the search problem. Based on the input key-value pairs, we generate a score between 0-k for every variant inside the project. We then sorting the variants in descending order of the generated score and pick the first k variants.

Time Complexity: O(n * m)

Example

Let us start with the 4 variants we looked at in part 1.

1. age group: 30-45, gender: male, location: London, purchases: 5+

2. age group: 20-30, gender: male, location: Delhi, purchases: 2-5

3. age group: 20-30, gender: female, location: San Francisco, purchases: 5+

4. age group: <20, gender: male, location: Sydney, purchases: 0-2

Let us say we are searching for a creative with gender: male and purchases: 0-2. We start with a base score of 0 for all the variants. We match the given key value pairs with each variant’s metadata. Each match is 1 point.

1. Variant 1: Gets 1 point as it matches gender: male

2. Variant 2: Gets 1 point as it matches gender: male

3. Variant 3: Gets 0 point as it does not match anything

4. Variant 4: Gets 2 points as it matches gender: male and purchases: 0-2

When we sort by the score, variant 4 comes out on top with a score of 2.

JS implementation

System Design

There are 2 parts to this architecture:-

1. Rendering - Creatives are rendered using a combination of Lambda (serverless) and auto-scaling EC2. They are stored in S3 with the URL saved in a MongoDB database.

2. Delivery - The API uses CloudFront, load balancer, Lambda, and Redis.

CloudFront

1. Is integrated with Lambda using Load Balancer as there is no direct way to connect Lambda and CloudFront

2. Is used to cache JSON response and/or binary creative data

3. JSON responses are cached against query parameters (for sending search attributes) and headers (for authentication)

4. Checks for same request in the nearest edge location, reducing overall latency drastically

5. Changes made to project in Rocketium triggers an invalidation request in CloudFront distribution so users always have the latest content

Load balancer

1. Just acts as a middleware to connect Lambda and CloudFront

2. Mapped as an origin inside respective CloudFront distribution

3. A behaviour needs to be defined for endpoint /api/v1/creatives to redirect request to mapped load balancer

Lambda

1. Avoids having to manage dedicated servers and scaling

2. Uses search algorithm defined above to find the most relevant creatives

3. Uses reserved concurrency to save cold start time and improve latency

Redis

1. Data retrieval is much faster from Redis than a database

2. In case of miss, variant metadata is fetched from MongoDB and cached in Redis

3. Lambda checks for a hit in Redis first before making a database call

Conclusion

In these two parts, we learned about the why, how, and what of Smart Links. Coming soon is Cortex, a layer on Smart Link which harnesses its power to run personalised in-app campaigns. Until then, see it in action at https://rocketium.com/is/in-app-campaigns/.

Need of personalized visuals at scale

Contrary to what social and news media suggests, we are truly living in an era of unprecedented abundance. Across every metric, humanity has never had so much of so many things. There is one thing that was limited and continues to be limited for the foreseeable future - time.

This makes attention a very precious commodity, especially in this digital era. Today, it is easier than ever to get in front of someone but not as easy to hold their attention. We are swamped with apps, emails, calls, ads across an ever-growing inventory of products and brands. Standing out from the noise is essential to capture someone’s attention. Only brands that capture attention have a shot at a deeper relationship with users.

We have seen the highest level of engagement when this happens:-

1. Users already have a relationship with the brand as users of its app

2. The app has visually-appealing banners and videos

3. These visuals are personalized by user preference, location, …

Troubles with today’s publishing process

Rocketium helps campaign teams run more effective visual campaigns faster. This involves creative design, management, collaboration, review, and more throughout the four stages that every campaign goes through:-

1. Plan

2. Create

3. Publish

4. Analyse

The publishing stage is a critical part of this lifecycle where creatives are delivered to ad platforms, marketing automation software, and, increasingly, apps and websites.

Today’s publishing process is extremely cumbersome. This summary should tell you how much worse this is in reality:-

• Designers share banners and videos with marketers as giant zip files

• Marketers pass these along to operations or product teams

• Operations or product teams upload these files one by one in an internal tool

• Rinse and repeat every time offers or prices change (typically, once in 2 days)

Our solution - Smart Links

Smart Links are an elegant way to bridge the gap from creation to handoff to delivery to analytics. They are a unique live link (URL) for a project (collection of creatives) that can deliver personalised creatives at lightning-fast speeds.

There is more to Smart Links than meets the eye.

• Search - A project can have hundreds or thousands of creatives in them. Each creative can have any number of tags associated with them as key-value pairs (for example, preference: vegetarian, type: premium). Smart Links have a search mechanism to choose the best-matching content given a set of tags. More on this in part 2.

• Real-time updates - Any updates made to the creatives on Rocketium are immediately visible to users. This eliminates the multi-step publishing process every time offers, prices, or messaging changes.

• Embed or programmatic use - Smart Links can be used directly in apps, websites, or other software just like image or video links making this a nice drop-in replacement for internal CDN links. With another query parameter, they can also return a structured JSON response. This allows deeper integration with marketing automation software or internal content management systems.

Smart Links in action

Preparing data for personalization

Let’s dive deeper into in the world of personalisation with Smart Links using an example.

Alice is a Category Manager at an e-commerce giant called Galaxy. She handles clothing department and oversees the process of delivering personalised banners for apparels on the landing page across the app and website.

Galaxy’s biggest sale of the year is during holiday season and Alice wants to make it a big success. Personalisation has been the cornerstone of Alice’s success in past campaigns and she wants to repeat it in this sale by reaching as many users as possible with highly personalised content.

She wants to start simple by personalising on four attributes:-

• Age

• Gender

• Location

• # past purchases

This means a 35-year old man from New York might see English creatives with blazer offers in English while a 18-year old girl from Paris might see French creatives with sneaker offers. The discount percentage each of them sees will depend on the number of past purchases.

Alice along with Galaxy’s designers and brand team plan the design language, layouts, experiments, and personalization points for the campaign. Designers make a base design template (aka key visual) that has a headline, offer, product, and a call-to-action (CTA).

In parallel, Alice plans all the user segments for whom she wants to personalize her campaign. These are combinations of age, gender, location, and number of purchases. Below are a few examples.

1. 30-45 years, male, London, 5+ purchases —> Dress shirts, English, 30% discount

2. 20-30 years, male, Delhi, 2-5 purchases —> Trousers, Hindi, 40% discount

3. 20-30 years, female, San Francisco, 5+ purchases —> Summer dress, English, 30% discount

4. <20 years, male, Sydney, 0-2 purchases —> T-shirts, English, 50% discount

She creates a spreadsheet with rows for each combination containing the above metadata and the content she wants to personalize for each row - headline, image from the website / internal content management system, CTA, and so on. She can create dozens or thousands of rows based on her needs.

Getting the personalized creatives

Alice logs into Rocketium, chooses the template that the Design team made, creates a project from it, and imports her spreadsheet into it. This replaces the placeholders in the template with spreadsheet data to create a new variant for each row.

Alice exports the project to render these creatives instantly.

Using Smart Links

Exporting a project automatically generates a Smart Link for the entire project. A Smart Link URL is of this format:-

https://cdn.rocketium.com/api/v1/creatives/project183hdj

To see full specs visit our API documentation.

Alice can hand over this URL to the Operations or Product team. Each individual creative within the project can be retrieved in milliseconds by passing the right attributes for it. For example:-

https://cdn.rocketium.com/api/v1/creatives/project183hdj?gender=male&location=sydney

As you might have noticed, you do not need to pass all the parameters to fetch the right creative. Smart Link is smart enough to return the best-matching creative based on the parameters. Oh yeah, and it is case-insensitive.

And here you go! Just like that, this holiday sale became a huge milestone for Galaxy and Alice.

Benefits of Smart Links

• Instantly update offers, make edits, or correct errors without deployment or further hand-off.

• Personalize emails and push notifications by replacing static images or GIFs with dynamic Smart Links. Smart Links integrates well with customer engagement tools like Braze, CleverTap, and Iterable.

• Add or remove personalization attributes as needed by modifying creative metadata. No technical or design effort needed.

Up next

Things are about to get technical. We’ll take a deeper look at what runs behind a simple yet powerful Smart Link URL in part 2 of this series.

The birth of our Creative Evaluation Engine - TARS

Over 200K+ creatives were generated on Rocketium in the last year. All of these have been from some of the highest-performing enterprises in our ecosystem. And we have been a quiet force behind their large-scale projects.

Sensitive information from campaigns is highly protected on the Rocketium platform. However, there is a hoard of data from successful, large-scale businesses who make creatives for different objectives (such as ads, app, push). And if we are as committed to our mission statement, we couldn’t have let this goldmine of data not be used for smarter, better projects.

Thus, TARS came into being. Just like its namesake, Rocketium’s TARS is a strongly tested and trained logic that uses the collective intelligence from the entire platform, looking at each element objectively, to help build high-performing creatives for our users. The engine enriches and unlocks existing data on the platform to make recommendations in the design for upcoming projects. And with every new creative made on the platform, the information is enriched further and it helps every new and old user to make better decisions the next time.

The vision: smart recommendations for better performing designs.

The whole project relied on two modules coming together in perfect harmony:

Tesseract: Building an analytics engine that can store and perform queries to retrieve ‘interesting’ information and insights from a large number of creatives, and then compare this information.

Tars: Building a recommendation engine that can compare new information (from creatives and user actions) with guidelines set by the company.

The final outcome would be the compilation of data from the analytics engine, and then providing it as an input to the recommendation engine for better performance of every design created.

Part 1 - Building Tesseract: The Analytics Engine

At Rocketium, we store the input data in a JSON format, and then it converts to a richer output. Our technical terminology for this codification of creatives would be JSON -> Video/Image/GIF converter. However, building this analytics engine cam with a few challenges:

Problem #1
Using MongoDB on our backend, the deeply nested creative data would be quite difficult to run analytics on. The existing data is quite heavy, and complex aggregated queries would be very slow in returning with a response.
So we had to figure out the best database, or some other analytics engine that can give better performance data.
We will circle back to this issue, but there’s more to look into for right now.

Problem #2

We wanted to slice and dice the data based on many keys, which meant we couldn’t use standard MySQL or MongoDB setups.
So we started with the research, and heavy as it was, we settled on Google’s BigQuery for the following reasons :
It allows for SQL-type data, but does not have the concept of indexes. This works in our favor, resolving the second problem mentioned above.
The performance was better once the data gets into the data flow.
It is quite cost-efficient when compared to other TTL tools such as Stitch - which we loved initially, but decided against for maintainability and control reasons.

Problem #3

If we were going to use two different databases to run this operation, we’d have to figure out a way to keep them synced.
To resolve this, we looked into the support from Mongo. We tackled this issue in two parts.

We needed to address the past data and then take a full dump. Our data was about 30GB or so, and the transform script must run at each level. However, ours was only running on a single portion of that. But we figured this was good enough to be POC, and we set out.

For keeping things in sync during runtime, we leveraged the watch command on Mongo to receive any changes to our code.

const tail = () => {
   const taskCollection = db.collection('capsules');
   const changeStream = taskCollection.watch({fullDocument: 'updateLookup', readPreference: 'secondary'});

   changeStream.on('change', (change) => {
      console.log(change.fullDocument);
   });
};

So finally, our data arrived into BigQuery in a flattened format. Our single table from Mongo was now four different tables in BigQuery. And the next step would be to form queries and show them via our UI, for actual user insights.

• You have exceeded the recommended ratio of the logo to the container. Consider resizing.

• The image to container ratio is exceeding the guideline for optimal CTRs in your category. Consider rearranging.

• Too many font styles are used. Legibility for your aspect ratio is not optimal.

As a newbie to this analytics team, I needed some time to recall what should happen next. But then I did, from something we all probably learned in XI grade. I identified that we can get histogram quantiles, and then make deductions according to the present value of inputs from a user. 

Sample query for the first statement regarding logo:

select 
  approx_quantiles(total_area, 100) as percentile 
from 
  (
    select 
      sum(
        if (
          stylesWithArea.area >= 100, 0, stylesWithArea.area
        )
      ) as total_area 
    from 
      (
        select 
          *, 
          (
            (
              (
                (styles.size_height / 100) * (styles.size_width / 100) * (
                  (
                    100 - styles.modifyLength_top - styles.modifyLength_bottom
                  ) * (
                    100 - styles.modifyLength_left - styles.modifyLength_right
                  )
                )
              ) / (
                styles.size_width * styles.size_height
              )
            ) * 100
          ) as area 
        from 
          `new-vidgen-servers.analytics.styles_dev` as styles 
        where 
          styles.elementType in ('image') 
          and LOWER(styles.helpText) like '%logo%'
      ) as stylesWithArea 
      left outer join `new-vidgen-servers.analytics.capsules_dev` as capsules on stylesWithArea.capsuleId = capsules.capsuleId 
      and stylesWithArea.version = capsules.version 
    where 
      capsules.outputFormat = 'image' 
    group by 
      stylesWithArea.capsuleId, 
      stylesWithArea.version, 
      stylesWithArea.size
  )

Now we had data in the format of 0 to 100 percentile, and what is the value at each number. We simply looked at the input value from the creative, and deduced the historical data to say something like:

83% of creatives have the logo, more than 2% of your creatives.

The entire analytics was executed at two levels, across the Rocketium data as well as within the customers’ scope. Tactically, we stored the entire query in our database with variables in the query such as {{teamId}} for providing dynamic values.

Part 2 - Building Tars: The Creative Evaluation Engine

I thought of building Tars the way that ESLint rules work in our React app. A user provides a config JSON, and while developing we get errors or warnings if there are any issues. Of course, a designer cannot know the full depth of the brand guidelines, especially if there is a large team with many shareholders. So our intention here was to resolve the issue of reviewing repetitive stuff altogether.

This meant providing suggestions like

The text should be between X and Y % of the banner.

The total number of characters in the banner between x — y.

Steps to building Tars

Having Tesseract in place was definitely a huge advantage, as we decided upon using the same logic to get the flattened structure for Tars as well. With JSON rules engine (https://github.com/CacheControl/json-rules-engine) as the base, we were going to build on top of it.

The modular library offered us a huge deal of liberty in what we were trying to achieve. A single rule would give us a yes or no, but we wanted to have a loop of these rules constantly in motion. Furthermore, we must collect the values on which elements break. 

We also needed aggregation in certain places, and I thought of the whole thing as if an SQL query. I had to build logic like grouping, filtering, and collecting data -- here is an example. 

{
   "_id" : "totalTextArea",
   "name" : "Total text area",
   "message" : "Text should be between 30 and 50 % of the banner",
   "ownerId" : "561796c9ce0203f33fe8e565",
   "order" : 1.0,
   "rules" : [
   {
      "type" : "filter",
      "key" : "styles",
      "outputKey" : "textElements",
      "customFact" : "area",
      "ruleJson" : {
         "all" : [
            {
               "any" : [
                  {
                     "fact" : "elementType",
                     "operator" : "equal",
                     "value" : "text"
                  },
                  {
                     "fact" : "elementType",
                     "operator" : "equal",
                     "value" : ""
                  }
               ]
            },
            {
               "fact" : "area",
               "operator" : "greaterThan",
               "value" : 0.0
            }
         ]
      }
   },
   {
      "type" : "aggregate",
      "ruleJson" : {
         "any" : [
            {
               "fact" : "sum_value",
               "operator" : "greaterThan",
               "value" : 50.0,
               "params" : {
                  "array" : "textElements",
                  "key" : "area",
                  "groupBy" : "size",
                  "groupByVal" : "max"
               }
            },
            {
               "fact" : "sum_value",
               "operator" : "lessThan",
               "value" : 30.0,
               "params" : {
                  "array" : "textElements",
                  "key" : "area",
                  "groupBy" : "size",
                  "groupByVal" : "min"
               }
            }
         ]
      }
   }
],
   "isDeleted" : false
}

How does this work?

If you closely go through the JSON above, it reveals the following:

• The flattened structure includes capsule, scenes, elements, and styles as the four base structures.

• The library compared data by determining greater than or less than values.

• Custom facts are used to calculate areas, or min/max values.

• Custom facts also come with a bunch of parameters.

• We have two methods for filtering, and then aggregating the data.

And finally, the rule engine was ready to use; all set to help customers define their rules.

To be continued

Everything incredible in the world has a version 2 which truly defines its capabilities. And Tars is going to follow the same route. We are yet to achieve all that we know we can offer, and we know what we want to achieve as well:

• Users making their own rules (for now we only offer them what we have put together in our database)

• Adding more complex rules to enrich users’ experience with more capabilities.

• Constantly checking our analytics engine and building more queries.

So we’re setting out on a bigger horizon. Until we get there and I share it with you, I hope this can suffice.

What’s the wait for — TRY IT OUT!

Go to https://rocketium.com and signup to see the mammoth Tars in action!