Maybe you searched "meeting link" in your work server. Or tried to find that meme someone shared 2 years ago. Or looked for a conversation with a friend from months back.

That simple search bar is querying trillions of messages across millions of servers. And recently, Discord completely rebuilt how it works.

Let me break down what happened.

First, Some Context

What happens when you search on Discord?

When you type "project deadline" in a Discord server:

1. Your query hits Discord's search infrastructure

2. It finds the right partition (called an "index") for that server

3. Scans through all messages to find matches

4. Returns results sorted by relevance

Simple enough. But now imagine doing this for:

• 200+ million monthly active users

• Millions of servers

• Some servers with billions of messages

That's a different beast entirely.

What is Elasticsearch and Lucene?

Discord uses Elasticsearch for search. But what is it?

Think of it like this:

• Lucene is the core search engine - the actual code that scans through text, builds search indexes, and finds matching documents. It's like the engine of a car.

• Elasticsearch is a wrapper around Lucene that makes it distributed and scalable. It handles splitting data across multiple servers, replication, and coordination. It's like the car itself - steering wheel, transmission, everything that makes the engine usable.

When you search on Discord, Elasticsearch receives your query and uses Lucene under the hood to find matches.

What is "indexing"?

Every message you send on Discord needs to be indexed - converted into a searchable format.

Raw message: "Hey, are we meeting tomorrow at 3pm?"

Indexed format: The words "hey", "meeting", "tomorrow", "3pm" get stored in a special data structure that makes them fast to search.

Without indexing, Discord would have to scan every single message character by character. With indexing, it can jump directly to messages containing your search terms.

What is "bulk indexing"?

Discord doesn't index messages one by one. That would be painfully slow.

Instead, they collect messages in batches (say, 50 at a time) and index them together. This is bulk indexing.

Analogy:

• ❌ Going to the post office 50 times for 50 letters

• ✅ Going once with all 50 letters

Way more efficient.

How Discord's Old System Worked

Here's the flow when you sent a message:

You send message
      ↓
Message enters Redis queue (waiting line)
      ↓
Worker picks up 50 messages from queue
      ↓
Worker bulk-indexes all 50 to Elasticsearch
      ↓
Messages stored across different nodes

Discord had 2 large Elasticsearch clusters with 200+ nodes each. Different Discord servers lived on different nodes.

This worked fine for years. Until it didn't.

The Problems

Problem 1: One Bad Node = 40% Failure Rate

Here's where things got ugly.

When a worker picked up 50 messages, those messages belonged to different Discord servers. So they went to different Elasticsearch nodes:

Batch of 50 messages:
├── Message 1 (Gaming server)    → Node 23
├── Message 2 (Study group)      → Node 87
├── Message 3 (Friend's DM)      → Node 45
├── Message 4 (Work server)      → Node 12
└── ... spread across ~50 nodes

Now, what happens if Node 45 crashes?

Elasticsearch marks the entire batch as failed. All 50 messages go back to the queue to retry.

Even though 49 out of 50 operations succeeded, one failure meant everything retried.

Let's do the math:

• Discord had ~100 nodes

• Batch touches ~50 random nodes

• Probability of at least one node being down ≈ 40%

So a single node failure caused 40% of all indexing operations to fail. Those failed batches went back to the queue, creating more load, causing more failures. A vicious cycle.

Problem 2: Redis Dropped Messages Permanently

Because so many batches were failing and retrying, the Redis queue got massively backed up.

Redis is an in-memory database. It's incredibly fast, but it has limits. When millions of messages piled up:

1. Redis CPU maxed out trying to manage the queue

2. To survive, Redis started dropping messages

3. Those messages were lost forever

The result? Some messages were never indexed. Users would search for something they definitely said, and it wouldn't show up. Because it was never made searchable.

Problem 3: Lucene's 2 Billion Document Limit

Remember Lucene, the search engine under Elasticsearch?

It has a hard limit: maximum 2 billion documents per index.

This sounds like a lot. But Discord servers like:

• Fortnite Official

• Minecraft

• Large gaming/streaming communities

These servers have billions of messages. They hit the limit.

When that happened, new messages in that server stopped being indexed. Search completely broke for their largest communities.

Discord's temporary fix? Work with the Safety team to find spam servers and delete them to free up space. Not exactly a scalable solution.

Problem 4: Couldn't Do Software Updates

Because the system was so fragile (one node down = 40% failure), Discord couldn't do rolling restarts.

Normally, you update software by:

1. Take down Node 1, update it, bring it back

2. Take down Node 2, update it, bring it back

3. Repeat for all nodes

But with Discord's system, taking down even one node caused chaos.

When the Log4Shell vulnerability hit (a critical security issue affecting most Java applications), Discord had to take the entire search system offline to patch it.

Other companies patched in minutes. Discord needed a maintenance window.

Problem 5: No Cross-DM Search

This one affected users directly.

Messages were organized (sharded) by server or DM channel. All messages from "Gaming Server" lived together. All messages from your DM with Alex lived together.

But what if you wanted to search across all your DMs?

"Where's that flight confirmation someone sent me?"

Discord would need to query every single DM channel you've ever had. If you've messaged 200 people, that's 200 separate queries. Too slow and expensive.

So cross-DM search simply didn't exist.

The Solutions

Discord didn't just patch these issues. They rebuilt the entire system.

Solution 1: Cell Architecture (40 Small Clusters)

Old: 2 massive clusters with 200+ nodes each

New: 40 smaller clusters grouped into "cells"

Each small cluster has:

• 3 Master nodes - coordinate the cluster (one per availability zone)

• 3+ Ingest nodes - receive and preprocess data (stateless, can scale up/down)

• Multiple Data nodes - store the actual messages (replicated across zones)

Why is this better?

If one cluster has problems, 39 others are completely unaffected. The blast radius of any failure is now much smaller.

It's like having 40 small libraries instead of 2 giant ones. One library floods? The other 39 still work fine.

Solution 2: Smart Batching by Destination

Old: Batch 50 random messages → send to 50 different nodes → one failure kills the batch

New: Group messages by destination before batching

Messages for Node 23: [msg1, msg4, msg12, msg38] → Batch A
Messages for Node 45: [msg2, msg7, msg29]        → Batch B
Messages for Node 87: [msg3, msg15, msg41, msg50] → Batch C

Now if Node 45 crashes:

• Batch A ✅ succeeds

• Batch B ❌ fails and retries

• Batch C ✅ succeeds

One node failure only affects that node's messages. Not 40% of everything.

Discord built a "Message Router" in Rust that:

1. Streams messages from the queue

2. Groups them by destination (cluster + index)

3. Spawns a separate task for each destination

4. Each task collects and bulk-indexes its own messages

Solution 3: Redis → Google PubSub

Old: Redis queue drops messages when overwhelmed

New: Google PubSub with guaranteed delivery

PubSub is a managed message queue service. Its key property: messages are never dropped.

If indexing slows down, messages wait in the queue. They might wait for minutes or hours. But they're never lost.

The tradeoff: during failures, search results might be delayed. But they'll never be permanently missing.

Solution 4: BFGs Get Dedicated Clusters

BFG = Big Freaking Guilds (Discord's internal term 😄)

For servers approaching the 2 billion limit, Discord now:

1. Creates a dedicated cluster just for that server

2. Uses multiple shards instead of one (spreads data across multiple Lucene indexes)

3. Migrates messages with zero downtime

The migration flow:

Step 1: Create new index with 2x shards
Step 2: Dual-write (new messages go to OLD and NEW index)
Step 3: Backfill historical messages to NEW index
Step 4: Switch queries to NEW index
Step 5: Stop writing to OLD index
Step 6: Clean up OLD index

Users notice nothing. Search keeps working throughout.

This is similar to how Stripe does zero-downtime database migrations - dual-write, backfill, switch, cleanup.

Solution 5: DMs Sharded by User

Old: DMs sharded by channel

• DM with Alex → Index A

• DM with Jordan → Index B

• DM with Sam → Index C

Searching across all DMs = querying A + B + C + hundreds more

New: DMs sharded by user

• All YOUR DMs → Your personal index

Searching across all DMs = querying ONE index

This required re-indexing every DM message. But since Discord was already rebuilding everything, they took the opportunity.

Now you can search "flight confirmation" and find it instantly, no matter who sent it.

Key Takeaways

1. Reduce blast radius

The old system had a tiny blast radius for each failure (one node), but the impact was massive (40% of operations). The new system contains failures to small, isolated areas.

2. Never drop data

Switching from Redis to PubSub meant accepting slower recovery for guaranteed durability. That's almost always the right tradeoff.

3. Batch intelligently

Batching is great for performance. But batch by destination, not randomly. This prevents one bad destination from affecting everything.

4. Plan for outliers

Most Discord servers are small. But the huge ones (BFGs) need special handling. Design your system to handle the 99% efficiently while having escape hatches for the 1%.

5. Migrations can enable features

Discord had to re-index everything anyway. They used that opportunity to re-shard DMs by user, enabling a completely new feature (cross-DM search).

Wrapping Up

Discord's search infrastructure went from "barely holding together" to "trillions of messages, 5x faster."

The key insight: the best systems aren't ones that never fail. They're ones where failure doesn't cascade.

One node failing should be that node's problem. Not a 40% catastrophe.

That's resilience engineering.

Full technical details: Discord Engineering Blog

1. AI (Artificial Intelligence).

   

   Artificial Intelligence is a field of computer science that focuses on creating systems capable of performing tasks that normally require human intelligence. These tasks include learning, reasoning, problem-solving, understanding natural language, and perception.

   Some common examples of AI include:

   1. Virtual Assistants: AI-powered assistants like Siri, Alexa, and Google Assistant help users with tasks like setting reminders, playing music, and answering questions.

   2. Recommendation Systems: Platforms like Netflix, Amazon, and YouTube use AI to suggest content based on user preferences and behavior.

   3. Self-Driving Cars: Companies like Tesla and Waymo are developing autonomous vehicles that use AI to navigate and make driving decisions.

   4. Healthcare: AI is used in medical diagnosis, personalized treatment plans, and even in robotic surgeries.

   5. Chatbots: AI chatbots like ChatGPT, Claude AI, and V0.dev can engage in conversations, answer queries, and provide customer support.

   6. Fraud Detection: Banks and financial institutions use AI to detect and prevent fraudulent activities by analyzing transaction patterns.

If you're interested in upskilling yourself in AI, I want to share a great GitHub repository that includes tons of resources:

• https://github.com/mrsaeeddev/free-ai-resources?tab=readme-ov-file

Want to share some other links as well which are really great if u want to learn more about AI and deep dive into it

• https://www.deeplearning.ai/courses/

• https://cloud.google.com/learn/training/machinelearning-ai

2. Cloud Computing and DevOps:

   

   Cloud computing and DevOps are crucial in today's tech landscape. Cloud computing allows businesses to store and access data over the internet, providing scalability, flexibility, and cost savings. DevOps, on the other hand, is a set of practices that combines software development (Dev) and IT operations (Ops) to shorten the development lifecycle and deliver high-quality software continuously.

   Why is it important?

   1. Efficiency: Cloud computing and DevOps streamline workflows, making it easier to manage resources and deploy applications quickly.

   2. Scalability: Businesses can scale their operations up or down based on demand without investing in physical infrastructure.

   3. Collaboration: DevOps fosters a culture of collaboration between development and operations teams, leading to better communication and faster problem-solving.

   4. Cost Savings: By using cloud services, companies can reduce the costs associated with maintaining physical servers and data centers.

How to improve it?

1. Automation: Implement automation tools for continuous integration and continuous deployment (CI/CD) to reduce manual errors and speed up processes.

2. Monitoring and Logging: Use monitoring tools to keep track of system performance and logs to diagnose issues quickly.

3. Security: Integrate security practices into the DevOps pipeline to ensure that applications are secure from the start.

4. Training: Invest in training for your team to keep them updated with the latest tools and best practices in cloud computing and DevOps.

Resources:

• AWS Training and Certification

• Google Cloud Training

• Azure DevOps Documentation

• DevOps Handbook

3. Full Stack Development

   

   Full stack development involves working on both the front-end and back-end parts of a web application. This means a full stack developer can handle everything from designing user interfaces to managing databases and server configurations.

   Why is Full Stack Development Important?

   1. Versatility: Full stack developers can work on various parts of a project, making them highly versatile and valuable to any team.

   2. Cost-Effective: Hiring a full stack developer can be more cost-effective than hiring separate front-end and back-end developers.

   3. Better Collaboration: Full stack developers understand the entire project, which leads to better communication and collaboration within the team.

   4. Faster Development: With the ability to handle multiple tasks, full stack developers can speed up the development process.

Tech Stack Examples:

1. MERN Stack: MongoDB, Express.js, React, Node.js

2. MEAN Stack: MongoDB, Express.js, Angular, Node.js

3. LAMP Stack: Linux, Apache, MySQL, PHP

4. Django Stack: Python, Django, PostgreSQL

Resources:

1. freeCodeCamp

2. MDN Web Docs

3. Codecademy

4. Coursera

5. Full Stack Open

4. Data science and Analytics

   

   Data science and analytics are crucial in today's data-driven world. They help organizations make informed decisions by analyzing large sets of data to uncover patterns, trends, and insights. Here are some reasons why data science and analytics are important:

   • Informed Decision-Making: Data science enables businesses to make decisions based on data rather than intuition. For example, a retail company can analyze customer purchase data to determine which products are most popular and adjust their inventory accordingly.

   • Improved Efficiency: By analyzing operational data, companies can identify bottlenecks and inefficiencies in their processes. For instance, a manufacturing firm can use data analytics to optimize their production line, reducing downtime and increasing output.

   • Personalized Customer Experiences: Data science allows companies to tailor their products and services to individual customer preferences. For example, streaming services like Netflix use data analytics to recommend shows and movies based on a user's viewing history.

   • Competitive Advantage: Companies that leverage data science can gain a competitive edge by identifying market trends and consumer behavior before their competitors. For example, financial institutions use data analytics to predict stock market trends and make strategic investment decisions.

Here are some popular data science tools:

• Python: A versatile programming language widely used for data analysis and machine learning.

• R: A statistical programming language that is great for data manipulation and visualization.

• Jupyter Notebooks: An open-source web application that allows you to create and share documents containing live code, equations, visualizations, and narrative text.

• Pandas: A Python library providing high-performance, easy-to-use data structures and data analysis tools.

• TensorFlow: An open-source platform for machine learning developed by Google.

• Tableau: A powerful data visualization tool that helps in creating interactive and shareable dashboards.

• Apache Spark: An open-source unified analytics engine for large-scale data processing.

• SQL: A standard language for managing and manipulating databases.

• Hadoop: An open-source framework that allows for the distributed processing of large data sets across clusters of computers.

• Power BI: A business analytics service by Microsoft that provides interactive visualizations and business intelligence capabilities.

Resources:

• Coursera: Offers courses on data science and analytics from top universities and companies.

• Kaggle: A platform for data science competitions and a rich source of datasets.

• edX: Provides online courses from institutions like Harvard and MIT on data science.

• DataCamp: Offers interactive courses on data science and programming.

• Udacity: Provides nanodegree programs in data science and related fields.

5. Cyber security

   

   Cyber security is the practice of protecting systems, networks, and programs from digital attacks. These attacks are usually aimed at accessing, changing, or destroying sensitive information, extorting money from users, or interrupting normal business processes.

   Examples of Cyber Security Threats

   1. Phishing: Fraudulent attempts to obtain sensitive information by pretending to be a trustworthy entity in electronic communications.

   2. Malware: Malicious software designed to harm, exploit, or otherwise compromise a computer system.

   3. Ransomware: A type of malware that threatens to publish the victim's data or block access to it unless a ransom is paid.

   4. Denial-of-Service (DoS) Attacks: Attacks intended to shut down a machine or network, making it inaccessible to its intended users.

Why Cyber Security is Important

• Protection of Sensitive Data: Ensures that personal and business information remains confidential and secure.

• Prevention of Financial Loss: Protects against financial fraud and theft, which can have severe economic impacts.

• Maintaining Business Continuity: Helps in keeping business operations running smoothly by preventing disruptions caused by cyber attacks.

• Building Trust: Ensures that customers and clients feel safe doing business with you, knowing their data is protected.

Resources for Learning Cyber Security

• Coursera: Offers courses on cyber security from top universities and companies.

• Cybrary: Provides free and paid courses on various cyber security topics.

• edX: Offers online courses from institutions like MIT and Harvard on cyber security.

In conclusion, the tech landscape is rapidly evolving, and staying ahead requires continuous learning and adaptation. The top 5 IT skills in 2024—Artificial Intelligence, Cloud Computing and DevOps, Full Stack Development, Data Science and Analytics, and Cyber Security—are essential for anyone looking to thrive in the industry. By investing time in mastering these skills and utilizing the resources provided, you can position yourself for success in the ever-changing world of technology. Happy learning! 😇

Let's consider an example of an entrepreneur named Yash. He built a social media platform called "ConnectWithWorld."

His initial setup is as following:

1. Technology: He is using MERN stack for the backend and frontend.

2. Single Server: His application is hosted on single server.

3. Traffic: Initially the traffic is not much. Its low to moderate most of the times. So single server is easily able to handle the load right now.

One day he shared his application on Linkedin and Twitter. His post went viral and lots of users started coming to his social media platform. They started using it daily and posting lots of videos, threads etc.

One day what happened is the platform started getting slow. Because it was not able to handle the amount of users who were using the platform and posting stuff, it went really slow. So Yash started researching about it and found out about Scaling.

Approaches which he did to handle the slowness:

1. Vertical Scaling:

   1. Currently his application is running on a single server hosted on some cloud provider (AWS / GCP).

   2. He upgraded the server and bought more RAM, more CPU and more Storage to handle the load.

   3. With the upgrade, performance improved a lot and users were really happy with how fast the application was working.

   4. One day, something bad happened, because of the huge load and lots of users posting threads and videos, server hit the limit and all its storage was full, RAM was full and eventually server got shut down. This led to a very big downtime for the application and all the users were really frustrated because of this.

2. Horizontal Scaling:

   1. To ensure reliability and availability of the platform, he decided to do horizontal scaling.

   2. He bought more servers from the cloud provider (AWS / GCP) and deployed his application code in all those servers.

      1. Because of this, now there is no single point of failure and even if one application goes down, we have another server which can serve the requests now

   3. Yash setup one load balancer as well, because now we have multiple servers and multiple users who are using the platform. So we need to divide the requests to all the machines, so that there is no high load on a single machine at a given time.

After doing both vertical and horizontal scaling, finally Yash was able to make the platform stable and serve lots of users with good speed.

There are other things involved as well as even after horizontal scaling like load balancing, database replication / sharding, CDNs etc. We will discuss that in coming blogs.

I hope this blog was really helpful and you all learnt something new.

Thank you all for reading this blog. Cheers 🍻

• Yatharth Verma

Description

Introducing FormVibe - Create Beautiful Forms with Ease

In the digital era, forms are a vital tool for gathering information and engaging with users. However, designing visually appealing and user-friendly forms can be a daunting task that requires technical expertise and consumes valuable time. That's where FormVibe comes to the rescue.

Simplicity and flexibility are at the heart of FormVibe. With its intuitive drag-and-drop interface, creating stunning forms has never been easier. Select from a variety of form elements and arrange them according to your needs, all with complete customization options. Whether you're creating a basic contact form or a complex survey, FormVibe has you covered.

Choose from a wide range of form elements, including text fields, dates, phone numbers, file uploads, yes/no options, and rating scales. FormVibe's diverse collection of elements ensures that you can capture the desired data accurately, no matter what your form requirements are.

One of the key highlights of FormVibe is its collection of custom templates. These templates serve as a starting point for users, offering pre-designed forms tailored to specific purposes such as job applications or surveys.

Once a form is created, FormVibe provides users with a preview of how the form will appear to the end users. This feature allows creators to ensure that their forms look and function exactly as intended before sharing them with their target audience.

FormVibe takes it a step further by generating a shareable URL for each form, enabling users to seamlessly distribute their forms through various channels such as email, social media, or embedded on websites. By providing a convenient and accessible way to share forms, FormVibe empowers users to reach their intended respondents effortlessly.

To help users stay organized and informed, FormVibe offers a comprehensive dashboard that displays all form responses in one centralized location.

FormVibe goes beyond the conventional form-building experience by offering users the freedom to select the preferred form rendering type for their audience. By providing choices such as Airtable or Typeform.

Airtable Form Type Preview

Typeform Type Preview

Recognizing the importance of branding, FormVibe allows users to personalize their forms by adding their own banner and icon images. By incorporating these visual elements, users can reinforce their brand identity and create a cohesive experience for their respondents.

FormVibe doesn't stop at form creation and data collection; it aims to deliver an exceptional user experience throughout the entire form-building process. The platform's user-friendly interface and intuitive features make it accessible to both seasoned developers and those with limited technical expertise. FormVibe's commitment to improving the developer experience and user experience is evident in its streamlined workflows and extensive customization options, enabling users to create forms that are not only functional but also visually stunning.

In conclusion, FormVibe empowers users to create beautiful forms with ease.

How I Started This Project

I spent my first week thinking about the idea. Then the second week I started making designs of how my actual web app will look when users will be using it.

Spent three to five days on the designs in the second week.

You can find all my wireframes and designs for the project here at the below link

Figma link

Attaching a small screenshot of my Figma designs

Then after finalizing the designs, Mid 2nd week I started analysing all the tech stack which I will be using for this project. I finalized the following tech stack.

Tech Stack I Used

• Appwrite

• NextJS

• Tailwind CSS - for Styling

• React-DND - for drag and drop functionality

• Formik and yup - for forms and validation

• Lucide icons

• Shadcn UI Components

In the third week, I started developing the project. I started with the authentication first. It was a really good experience learning Appwrite and integrating it into the project.

After authentication, I started making my dashboard and builder page according to the designs. Following are some of the challenges I faced while making this web app.

Challenges Faced

Building FormVibe was an exhilarating journey filled with numerous challenges and learning opportunities. Throughout the development process, I encountered a series of obstacles that required careful analysis, innovative solutions, and the utilization of cutting-edge technologies.

• One of the initial challenges I faced was implementing drag-and-drop functionality. As an integral feature of FormVibe, it was crucial to find an efficient and reliable solution. After thorough research and experimentation, I discovered React DnD, a powerful library that enabled the seamless integration of drag-and-drop capabilities into the application. Leveraging this library, I successfully incorporated intuitive drag-and-drop interactions, allowing users to effortlessly create their customized forms within FormVibe.

• Another significant challenge I encountered revolved around rendering the forms and making them editable for customization. Achieving a dynamic and user-friendly interface was paramount. I delved into exploring various techniques and patterns to accomplish this goal. By leveraging React's component-based architecture and state management, I devised an ingenious solution that allowed users to manipulate and customize their forms with ease. The ability to add, edit, and remove form elements became a seamless and intuitive process, empowering users to tailor their forms to their unique needs.

• In addition to form creation, handling user responses was yet another obstacle I had to overcome. It was imperative to design a robust mechanism that would efficiently collect, store, and manage the data submitted by users.

• Furthermore, offering full customization of forms was a critical aspect of FormVibe. I strived to provide users with an extensive range of customization options, allowing them to create forms that aligned precisely with their branding and aesthetic preferences. Implementing this level of flexibility required meticulous planning and a comprehensive understanding of user requirements. By leveraging a combination of CSS frameworks, design libraries, and flexible component architectures, I enabled users to personalize their forms extensively, making FormVibe a versatile tool for a wide array of use cases.

Building FormVibe was undoubtedly an enjoyable and rewarding project, albeit filled with challenges that pushed the boundaries of my knowledge and skills.

How Appwrite Helped in this Project

As a developer who has explored various BaaS and cloud solutions, I can confidently say that Appwrite stands out from the crowd with its exceptional simplicity and user-friendly experience. Having used platforms like Firebase in the past, I have come to appreciate Appwrite's seamless integration process and intuitive console, making it my top choice for backend development.

During my journey with Appwrite, I had the opportunity to integrate various services, each of which proved to be a valuable addition to my projects.

• The Authentication service provided a secure and straightforward way to implement login and signup functionality, ensuring that user data remains protected. With just a few lines of code, I was able to incorporate robust authentication features into my applications, saving me valuable time and effort.

• Appwrite's Databases service offers a reliable and scalable solution for storing forms, user data, and responses. Its ease of use and powerful querying capabilities allowed me to efficiently manage and retrieve data, providing a seamless experience for both developers and end-users.

• When it came to storing files or images, Appwrite's Storage service proved to be a game-changer. Uploading and retrieving files was a breeze, thanks to the simple API integration and robust infrastructure behind the scenes.

In conclusion, Appwrite has undoubtedly revolutionized the way I approach backend development and integrate cloud services.

Code Repository Link

FormVibe is proudly an open-source project, welcoming contributions from the community.

If you're passionate about FormVibe and would like to contribute to its development, I encourage you to visit my GitHub repository at

FormVibe Github Repository

By exploring the repository, you can delve into the codebase, propose improvements, report issues, and even submit your pull requests. Your contributions will help shape the future of FormVibe and make it even more remarkable.

Together, let's make FormVibe an exceptional web application for creating beautiful forms with ease and efficiency.

Demo Recording of Project

You can watch a demo of my project on the below link. I have uploaded it to my youtube channel.

https://youtu.be/FRMfz-3pbi0

This was all about my project. It was fun building this project. Also got to learn a lot about Appwrite.

Do checkout FormVibe here:

https://form-vibe.vercel.app/

Test credentials which you can use to test the website

email: testuser@gmail.com

password: Test123@

Thank you hashnode and appwrite for this hackathon opportunity.

Let's first understand what exactly is package manager just in a brief.

Package Manager

Package manager lets you install and manage dependencies your javascript project requires to work. Before moving forward lets learn little history about npm how it came and all.

History

npm (Node Package Manager) is the first package manager. It was released in 2010. It comes prebuilt with NodeJS. You don't have to download it explicitly. Earlier before npm, developers had to manually download the dependencies and manage them in their projects. But it became difficult to manage them as projects grew. That's where npm came in and helped developers easily manage the dependencies under a single folder named node_modules in your project. Over time, the tool evolved to include features like versioning, dependency resolution, and a package publishing system.

In 2015, NPM was acquired by Joyent, the company that created Node.js. This acquisition brought additional resources to the NPM project and helped it become a critical part of the Node.js ecosystem.

In 2016, NPM faced a significant security incident where a malicious package was published to the repository. The package was able to steal sensitive information from developers who installed it, highlighting the importance of package security. Since then, NPM has implemented additional security measures, including two-factor authentication and automatic vulnerability scanning.

Why other package managers came in the market?

On October 2016, Facebook announced a collaborative effort with Google and a few others to develop a new package manager that would solve the issues with consistency, security, and performance problems that npm had at the time. They named the alternative Yarn, which stands for Yet Another Resource Negotiator.

Yarn's architectural design is almost similar to npm, but it's much faster than npm as it parallelly installs all the packages and their dependencies. But still from the disk efficiency parameter it's not good as its using the same underlying structure as npm.

That's where pnpm helps. Let's understand that in the next section.

Pnpm and its benefits

Version 1 of pnpm was released in 2017 by Zoltan Kochan. It is a drop-in replacement for npm, so if you have an npm project, you can use pnpm right away!

It solved the major problem that yarn and npm both face is the flat dependency structure in node_modules meaning it stores all the dependencies and interdependencies in node_modules in a flat hierarchy. The main problem with this is disk storage fills up as for every project flat node_modules will be created and also it takes time to install all of these again and again for every project

pnpm solves this problem using content addressable storage (single source of truth) where it keeps all the dependencies folders at a single place. It uses hard links and symbolic links. In node_modules pnpm only keeps the parent dependencies or the main dependencies which your project depends on and all the other interdependencies will be stored as symbolic links (which means a reference to the files in content addressable storage)

So because of all these pnpm installs packages much faster and uses less disk space in comparison to npm and yarn

All these were some differences between yarn, npm and pnpm. Let me know in the comments which one you use and what you like most about that package manager.

Have a nice day

Regards,

Yatharth Verma

Software Engineer

Do checkout my portfolio

In this thread we will learn about access control security for procedures. Recently I was working on one project in my company where I needed one procedure in the production database along with permissions to execute it with my custom user.

But I was not able to execute it. It was throwing an error access denied for some other user which I was not even using to connect to database and execute procedure.

Then I researched about this a lot. And found out that all stored programs like SQL Procedures, Views, Triggers and events they execute within Definer context. What does this mean ?

It means whoever creates the procedure, execution always happen in that user context. If that user does not have all the privileges for operations which procedure is performing, other users who are trying to invoke the procedure will also not be able to execute the procedure. It does not matter whether that user has full permission or not.

Two solutions are there to solve this

1. Either get the procedure created from root user who has all the privileges.

2. Second solution is to define SQL Security Invoker keyword while procedure creation

Example:

CREATE DEFINER = 'admin'@'localhost' PROCEDURE p2()
SQL SECURITY INVOKER
BEGIN
  UPDATE t1 SET counter = counter + 1;
END;

For learning about this in more details, I would recommend everyone to check the article here as well. It has everything in detail

Link: https://dev.mysql.com/doc/refman/5.6/en/stored-objects-security.html

Thank you all for reading this.

The two best solutions which are available online and can help import svgs properly are listed below. You can use any method listed below to import svgs properly and use in Nextjs

Solution 1

Use SVGR. Install it by running the below command

npm i @svgr/webpack

Add the following code in next.config.js file and you are all done

module.exports = {
  webpack(config) {
    config.module.rules.push({
      test: /\.svg$/,
      use: ["@svgr/webpack"]
    });

    return config;
  }
};

Restart the nextjs server, as next.config.js file is created / updated.

After that, you can use svgs without any issues. Below is one simple example to import svg and use in your code

import React from 'react';
import SVGBackground from "./Background.svg";

export const AppComponent = () => (
  <div>
    <SVGBackground />
  </div>
);

Solution 2

Use babel-plugin-react-svg. Install it first

npm install --save-dev babel-plugin-inline-react-svg

Create / Update .babelrc file in the root of your project with the below code

{
  "presets": ["next/babel"],
  "plugins": ["inline-react-svg"]
}

And you are done. You can now use svg as components in Nextjs properly without any hassle.

These two are my favourite methods to use svgs in Nextjs. There are other available methods also which can be found in the internet which you can also use.

Thanks for reading this article. I hope it was useful.

Keep learning and keep rocking 🚀

New built in functions or new syntaxes in javascript, they might not work in outdated browsers. Some javascript engines might not be supporting them. To solve this problem, polyfills comes into the picture.

Pollyfill is like a code snippet that helps browser to support javascript new or native functions as well as new syntaxes to work properly.

Lets discuss a very simple example.

Lets say in some outdated browsers Array.filter function is not working. So lets create a polyfill of filter function which will work in all the browsers which does not support filter function

Example Code:

    Array.prototype.filter = function (callbackFn) {
      let outputArr = [];

      this.forEach((el, index) => {
        if (callbackFn(el, index, this)) {
          outputArr.push(el);
      }
    });

    return outputArr;
  };

  console.log([1, 2, 4, 5, 6, 7].filter((el) => el % 2 == 0));

So in the above example, I am creating a polyfill of filter function. I am replacing the filter function present in prototype of Array with my own polyfill function which is doing following:

1. Creating a new array
2. Looping over the existing array and checking whether the condition is true or not. If condition is true, we are putting the elements in new array
3. Lastly we will return the array

In this way polyfill can be created.

Lets discuss one more example. Lets create a polyfill of map function

Example Code:

  Array.prototype.map = function (callbackFn) {
    let outputArr = [];

    this.forEach((element, index) => {
      outputArr.push(callbackFn(element, index, this));
    });

    return outputArr;
  };

  console.log([1, 2, 3, 4].map((e) => 10 * e));

Its almost similar to the first example. Only difference is we are not checking conditions. We are executing the callbackFn on every element and pushing the result of callback function in the new array that we will return.

So, I hope now you understand what are polyfills and how to create one. There are other examples of polyfill as well. For every existing function in javascript you can make a polyfill to make older or outdated browsers to support them.

Thank you all. Regards, Yatharth