To give some context, I am bad at hikes, like terrible. I don't know how to describe it, but I am slow, I need time to motivate myself in every step I take when I am hiking.

I never told this to anyone but I love mountains and there is a part of me that loves this entire concept of climbing mountains, soaking in the beauty of nature, and reaching the peak. But then comes this other side that I think for a long time has been dominating, the one where I constantly feel that I can't do it because maybe I am not physically fit or because I am slow when it comes to climbing mountains or the leg pain that happened to me 2 years back will be back again.

The same story happened this time as well. I agreed to join the trek with my team but one day before I started doubting the yes.

Anyway, I joined for the trek & there we were on the bus at 4 in the morning starting our journey to Savandurga. I was sleeping throughout the journey & didn't think about the hike.

As soon as we reached & got off the bus, I saw the mountain, my first reaction was How the hell am I going to climb this?Is there a way I can run from here? But none of this was an option & I started mentally preparing myself for the long journey ahead.

10 minutes into the hike & I had given up. I was like, I am sitting here and asked the rest of the folks to continue. Hari stayed back with me, & we slowly started our journey to the peak with enough breaks in between. As we were climbing up, I was blown away by the view. There was a feeling of calmness and gratefulness in being able to witness the view.

There were enough places in between where I had given up but thanks to the constant motivation and encouragement from Hari I somehow reached the Fort Wall. I had no plans on going to the peak and was done.

By this time, the rest of the folks had visited the peak & were on their way back to the Fort Wall. When they reached there, Sivanesh mentioned the peak was only 15 minutes from the Fort Wall & which motivated me, I have this moment, right now, and I have to make the most of it. I don't know if I will ever visit Savandurga again & wanted to prove to myself that I can do it and not go back home with the self-doubt and regret of not finishing it.

I mentally prepared myself & thus began the journey to the peak. It took me more than 15 minutes to reach the peak but when I reached there, it was the best feeling ever. Suddenly all the self-doubt that had been cropping up slowly vanished.

As I sat at the peak, slowly soaking in the view, I realized that sometimes, the things that scare us the most are probably the things that we need to do the most. All the things that I have been worried or scared about felt so small in front of the mountains.

Soon, we started our descent, and as a gift for completing the hike, Karthik plucked these flowers from the tree.

As we started our descent, I couldn't stop thinking about how at the very same points during the ascent I was under doubt on whether I would ever finish it or not.

This was my fourth hike, but Savandurga is so far my favorite one.

I never thought I would say this, but for some reason, I can't wait to do my next hike. I know there is still a lot of effort I need to put in to prepare myself but I am looking forward to the next one.

PS: Once again, thanks to Hari, Karthik, Sivanesh, Akshay, and everyone for being patient with me & helping me finish the hike.

But you know, there are some things about giving talks that not many people talk about. I'm not sure if it's just me feeling this way or if others go through it too.

Think of it like a coin – there are two sides to preparing for tech talks.

The Bright Side

Feeling Proud

Giving tech talks makes you feel proud. When you take complicated ideas and break them down into simple bits that people can understand, it's like winning a challenge. You're not just sharing what you know or giving back to the community; you're proving to yourself that you can overcome tough things. This makes you more confident and helps you grow personally.

Making Connections

One of the coolest things about tech talks is meeting awesome people. It's not just about standing on stage alone; it's about creating a community. The friendships you build during these talks go beyond job titles and company names. Networking is more than just a fancy word; it's like having a group of friends who support you.

The Shadow

Feeling the Pressure

Behind those fancy slides and well-spoken speeches, there's a hidden challenge. Speakers often feel a lot of pressure. The fear of talking in front of people and the expectation to be perfect, tailoring it to the audience to make sure everyone gets the most value out of the content can be really tough on the mind.

Doubting Yourself

Ever heard of imposter syndrome? It's that feeling of not being good enough, and it can bother even the smartest folks. The more people notice your tech talk, the more imposter syndrome might sneak in and make you doubt yourself.

In simple terms, giving tech talks is like a rollercoaster ride. There are highs where you feel proud and connected, and there are lows where you feel the pressure and doubt. It's a mixed bag, and I wonder if others feel this rollercoaster too.

In conclusion, despite the challenges that come with giving tech talks, I've found the experience to be incredibly rewarding. Each presentation has not only allowed me to share knowledge but has also revealed a new version of myself.

To anyone who reads this & plans on taking a first step towards giving their tech talk, remember that behind a perfectly delivered talk that you have witnessed is a speaker who was equally nervous at some point.

So, to those considering taking the stage, relish the opportunity to discover and present the ever-evolving you—one talk at a time.

Asking for a Review

1. Ensure Github Actions Pass: Before assigning a Pull Request (PR) for review, it's crucial to ensure that all Github actions have passed successfully. This basic requirement helps confirm that the new code doesn't break existing functionality and saves reviewers time.

2. Break Down the Code: For large Pull Requests, breaking down the code into multiple commits within the same PR can make the review process more manageable. This allows reviewers to focus on specific changes and understand them in isolation.

3. Detailed PR Description: Provide a comprehensive PR description, explaining what the changes are, why they are necessary, and the testing plans. A well-documented PR description helps reviewers understand the context and purpose of the changes.

Giving a Review

1. Gather Just Enough Context: In the initial review, focus on getting just enough context about the feature or issue without understanding the implementation from the author itself. This minimizes biases and increases the chances of identifying blind spots in the code.

2. Review the Code in Steps: Start by reviewing the entire code to get a high-level understanding of the changes. Then, dive deeper into specific functions or segments to identify performance optimizations, decoupling of code, and large functions.

3. Understand Testing & Deployment Plans: Before giving your approval, it's good to understand the testing plans for the feature. Ensure that the code has been adequately tested, covering both functionality and edge cases. At times when there are changes in different repositories that are dependent on each other or there is a certain production script to be run before merging the code, it's good to have an understanding of the plan to push the code to production.

4. Do a QA of the feature: Reviewing is more than just the code. Try testing the feature, validating the technical approach, and ensuring it meets product expectations.

5. Maintain a Priority List: I usually maintain a list of the PRs & the expected date of when the feature needs to go live. This helps me organize the reviews by priority. Next, when reviewing the code, understand which parts of the code are most critical and can lead to production issues. Those hot paths should be reviewed first. This approach helps ensure that the most important issues are addressed on priority.

6. Avoid Context Switch: To maintain focus and ensure thorough review, avoid reviewing multiple different features together in a single session. Context switching can lead to confusion and overlook critical issues.

7. Focus on Code Quality: Once the functionality is confirmed as correct, and any necessary performance optimizations are in place, comment on code quality. Address issues related to code structure, maintainability, and adherence to coding standards.

8. Suggest, Don't Instruct: When making suggestions or pointing out issues, use a constructive and supportive tone. Instead of instructing with phrases like "You should stop doing," opt for more collaborative language like "I think it would be better to." Avoid blameful or aggravating language.

As the reviewing engineer, remember that your approval is required before the code can be merged. This is a significant responsibility, as it ensures that the codebase remains robust and clean.

When I was a kid, I used to look forward to exam time because it gave me the justification I needed to go out and buy a new pen and all the stationery I wanted. The feel of the grip, the ease of the ink, and the elegance of the design all played a part in my decision when selecting the ideal pen; it became an art form. Each new pen I acquired served as both a tool and a companion, bearing witness to both my highs and lows. My favorite pens were Uniball ones. All of the pens I used in school are still safely stored.

Not only pens, but also board markers and diaries used to excite me, and they still do. I am happiest doing stationery shopping or when someone gifts me stationery.

Stationery, in my opinion, is a source of power. A pen has the ability to transform me into a person of authority, ready to conquer whatever comes my way. There's an undeniable allure to using a luxurious pen, whether it's during a meeting or simply in the solitude of journaling in my diary or creating my checklist. It's an elegant and sophisticated symbol that inspires me to embrace my thoughts and ideas. A pen provides an unparalleled sense of empowerment; it is more than just an instrument; it is an extension of myself. I recently purchased a set of ten different colored pens to make my note-taking more enjoyable. I carry it everywhere I go.

The same is true for whiteboards and markers. When I was in school and saw teachers writing on the whiteboard, I thought they were so powerful. By writing on the whiteboard, they had the power to transform the world. One of the reasons I wanted to be a teacher when I grew up was because of this. I got my first whiteboard in fourth or fifth grade where I used to solve maths problems. I still have a whiteboard which has now become my place to write checklists.

Every pen, marker, and diary serves as a link between my thoughts and the physical world. They motivate me to slow down and appreciate life's simple pleasures.

So, the next time you pass by a stationery store, stop for a moment and let the magic of stationery enchant you. With each pen stroke and page turn, you might discover a universe of enchantments waiting to be discovered—one colorful pen at a time.

Different Types of Index

B-Tree Index

This is the default type of Index in MySQL which is based on the B-Tree data structure.

The B-Tree index doesn't need a full table scan to find the desired data hence it speeds up the data access process. Similar to how a B-Tree works, in this index, the search starts from the root node and goes to the leaf node. Each node contains the minimum and maximum values for the child nodes. We find the right pointer by looking at these values and reaching the leaf node that has the pointer to the data needed. The depth of the B-Tree depends on how big our data is.

B-Tree stores the columns in an ordered manner which makes them useful for searching over a range of data, ORDER BY and GROUP BY operations.

B-Tree does a lookup from the leftmost indexed columns. Let's take an example to understand how adding a B-Tree Index will help us.

Consider the example of an Employee Database where we want to search for employees.

Employee Table Schema

CREATE TABLE EMPLOYEE (
    first_name varchar(40) not null,
    last_name varchar(40) not null,
    department varchar(40) not null,
    joining_date date not null
);

Consider the following data inserted in the table,

Now that we have the schema defined, let's try to search for some Employees in the table and understand how the indexes can be defined and used.

Single Column Index

ALTER TABLE EMPLOYEE (
    ADD INDEX first_name, 
    ADD INDEX department, 
    ADD INDEX joining_date
);

Query 1:

SELECT * FROM EMPLOYEE WHERE first_name = 'Donna' and department = 'Engineering'
-- Uses the Index on first_name

Even though there is an Index on the department column, we only use the index on first_name as the three individual indexes act as three separate tables.

Now when we run this query, MySQL will have to scan across 7 rows where the first_name is Donna, imagine the same scan across a huge table where around 1 million people have a similar name. Adding a single-column index won't help a lot.

Multi-Column Index:

ALTER TABLE EMPLOYEE (
    ADD INDEX (department, first_name, joining_date)
);

The following table contains a composite index on (department, first_name, joining_date).

Query 1:

SELECT * FROM EMPLOYEE WHERE department = 'Engineering' AND first_name = 'Donna'
-- Uses the Index on (department, first_name)

As compared to single column index, this will be faster as the number of rows to scan will get decreased by using the composite index on (department, first_name)

Query 2:

SELECT * FROM EMPLOYEE WHERE last_name = 'Hardy'
-- Doesn't use any Index

Query 3:

SELECT * FROM EMPLOYEE 
         WHERE first_name = 'Donna' 
         AND joining_date > 2010-07-28
-- Doesn't use any Index 
-- since the lookup follows the leftmost side of indexed columns
-- we are not mentioning the department
-- hence composite index is of no use

Query 4:

SELECT * FROM EMPLOYEE 
         WHERE department = 'Engineering' 
         AND first_name LIKE '%S%' 
         AND joining_date = 2010-07-28
-- Uses the Index on (department, first_name)
-- since for first_name we have a range query 
-- it will only use first two columns of index

In the case of using indexes for range queries, we should try using the range condition field last in the index. This is because after applying the range query there is no way for the B-Tree to apply the next filter, hence we should keep the range condition column at last.

For the Multi-column Indexes in this Table, we have index search capabilities on (department), (department, first_name) and (department, first_name, joining_date)

ORDER BY

Consider the following query

SELECT * FROM EMPLOYEE WHERE department = 'Engineering' ORDER BY last_name

This will use the department index to give us the records, and then perform a sort on the records to sort them by last_name. This is because we didn't have any index on last_name hence the records were not sorted by that column. This type of sort fetches the data in a temporary buffer before returning it (File Sort) which leads to extra computation.

This can be avoided by adding an index on last_name. The new composite index for us would be (department, first_name, joining_date, last_name)

GROUP BY

Proper Indexing also helps in GROUP BY operation.

Consider an example like

SELECT * FROM EMPLOYEE WHERE department = 'Engineering' GROUP BY first_name

The records would already be sorted department hence speeding up the filtering. After these results are returned they are also then sorted by first_name due to the second part of the index, and so they are already grouped for us. This won't require any other sorting hence making our query fast.

Index Selectivity

Index selectivity is the ratio of a number of distinct indexed values(cardinality) to the total number of rows in the table(#T). The range for selectivity is from 1/#T to 1.

An index is said to be highly selective if for each value we have fewer rows. Having a highly selective index is good as it filters out more rows when searching for query matches.

Choosing the Right Order

Now that we have understood what Index Selectivity is, let's see how it helps in deciding the column order.

Consider the same Employee Table Schema with the multicolumn index on (department, first_name, joining_date, last_name) . This composite index will perform better than a single column on queries but let's check if this is the perfect column order for the given set of values in the Employee Table.

Firstly, let's check the Index Selectivity of each of the four columns,

1. first_name: 3/9 = 0.33

2. Department: 4/9 = 0.44

3. last_name: 9/9 = 1

Since joining_date is a date field and we might want to do range queries on it, it is good to have it last in the order of index.

Based on the Selectivity, the ideal order looks like this (last_name, department, first_name, joining_date).

If we see the Query Examples above the initial order we defined works well for us as we can use Indexes for most queries. Hence along with selectivity, the use case of the WHERE clause also matters.

In Conclusion, the Index order for the Multi-Column Index depends on:

• WHERE conditions to effectively lookup data

• Use of ORDER BY and GROUP BY

• Use of Range conditions in the query

• Index Selectivity when unsure about the column order

• The columns we want to SELECT in the Query

It is important to note that the indexes shouldn't be optimized only for the WHERE clause but for the entire query.

Hash Index

Hash Indexes are used when we want to do exact lookups on every column in the table. Hash codes are generated for each row on the indexed columns. Since the original values are converted to short values based on the hash function, they are compact.

In the case of Hash Indexes, the index contains only pointers to the values rather than the entire values, hence MySQL uses the in-memory rows to fetch the values from the pointers.

Some of the drawbacks of Hash Indexes are

1. Since we are generating Hash using a function, we can't use it for partial matching of values. Example: If we have an Index on (A, B), we can't use this Index in a query that only uses column A.

2. In case of two or more rows having the same values for a column, when generating the Hash Index, it will lead to a collision. In such a case, the engine will traverse all the row pointers and read the values for comparison to find the right value leading to extra computation.

3. Hash Indexes are not very useful for columns that have low selectivity since they will lead to hash collisions.

4. They can't be used for sorting because they don't store rows in a sorted fashion.

Adaptive Hash Index

In the InnoDB storage engine when the values are being used frequently it creates a hash index on top of the B-Tree Index which leads to faster lookups. This is known as adaptive hash indexes.

Other Types of Indexes

Some other available Index Types include the Spatial Index which is used for geometry and geography types and Full-text Indexes which split the text into words and make an index over the words and not the whole text. This works faster for text searches when looking for specific words.

Benefits of Indexing

The benefits of indexing are measured on the three-star system

1. One Star if the Index reduces the rows that we have to scan.

2. Two Star if it improves performance by removing the overhead of sorting and temporary tables.

3. Three Star if all the required columns for querying are a part of the index.

Points to Remember while Indexing

Use of Functions in Queries

MySQL can use columns only if they are isolated in the query. We should avoid using functions in the WHERE clause.

Bad Query:

SELECT * FROM EMPLOYEE 
         WHERE department = 'Engineering' 
         AND YEAR(joining_date) = 2010

The above query will only use the index department even though an index is present on joining_date. This is because we have enclosed joining_date within a function.

Index Merge

In the case of multiple single-column indexes, MySQL sometimes uses multiple indexes by performing Index Merge. But it is not efficient since in the case of OR conditions, operations like sorting, grouping can utilize a lot of resources. In the case of AND conditions, it makes more sense to use a multicolumn index over multiple single-column indexes.

Other Things to Note

1. Indexes are implemented at the storage engine level and not the server level, hence differs from engine to engine.

2. Even though we use an ORM, we can't rely on it for indexing as they usually generate queries that satisfy the underlying logic.

3. Indexes come with drawbacks as well. There is a performance impact on INSERT, UPDATE, and DELETE operations. Every time we perform a write on a table, the indexes have to be maintained. Hence, it's important to choose Index wisely.

4. Always Test the indexes on the queries the application is performing by measuring the response time.

5. Using the EXPLAIN statement to understand how the query is executing and reveal why it is slow. Explain statement shows values such as the indexes used, order used for joining the table, rows accessed, use of Filesort or not, and a temporary table used or not.

I hope this article helps you in starting to use Indexes in your MySQL Databases.

In this blog post, we will delve into the world of web scraping with Python, exploring its definition, differences from web crawling, traditional methods, implementation with Python, how can the data be utilized, importance, and ethical considerations.

What is Web Scraping?

Web scraping refers to the automated extraction of data from websites. It involves parsing the HTML structure of web pages, extracting specific data elements, and storing them in a desired format, such as a csv or database. By automating the retrieval of data, web scraping saves time and effort compared to manual data collection.

Web Scraping Vs Web Crawling

Although the terms web scraping and web crawling are frequently used interchangeably, they are not the same thing.

Web scraping typically involves extracting specific data from targeted web pages, whereas web crawling entails traversing the web systematically in order to index or analyse web content.

A web crawler starts with a seed, which is a list of URLs to visit. The crawler finds links in the HTML for each URL, filters those links based on specific criteria, and then passes those links to a scraper so that the desired information can be extracted from them.

Web scraping is a subset of web crawling that serves more specific purposes such as obtaining product information, obtaining customer reviews, or gathering news articles.

Traditional Methods of Web Scraping

Before the libraries with Python came into picture, the go to methods to get data from the internet included:

1. Regular Expressions: To extract data from structured HTML documents, regular expressions were commonly used. We could use regex syntax to define patterns that would match specific data elements within the HTML source code. Regex-based scraping, while powerful, was limited to cases where the HTML structure was predictable and consistent. Handling complex or nested structures with regular expressions can be difficult and error-prone.

2. Manual Copying and Pasting: Manually copying and pasting data from websites into a local file or spreadsheet was one of the earliest and simplest methods of web scraping. This method worked well for scraping small amounts of data, but it became inefficient and time-consuming for larger-scale scraping tasks.

These traditional methods had drawbacks. They worked best with static websites with simple HTML structures and struggled with dynamic or JavaScript-rendered content. Furthermore, these methods were less scalable and necessitated a significant amount of manual labour, making them inefficient for large-scale data extraction tasks.

Web Scraping with Python

Some widely used Python libraries for web scraping include BeautifulSoup, Scrapy, Selenium & Extruct.

BeautifulSoup

BeautifulSoup is used for parsing HTML and XML documents. Beautiful doesn't directly interact with the server of the url we are trying to scrape. We need to use libraries like request to get the response data from the url. Once that is done, we can use parser like html.parser or lxml's parser to get the HTML content. Once we have the HTML content, we can fetch the required data.

We can use Beautiful when we need to extract data from a single webpage or webpages of the same HTML structure which don't require complex navigation. One drawback with BeautifulSoup is, it works only for static web pages.

import requests
from bs4 import BeautifulSoup

url = "https://beautifulsoup.com"
response = requests.get(url)

# Parse the HTML content
soup = BeautifulSoup(response.content, "html.parser")

# Extract data in the title tag from the parsed HTML
title = soup.title.text
print("Title of the page is ", title)

Selenium

Selenium is a web testing framework that allows automated browser interactions. It can be used for web scraping by controlling a browser instance and extracting data from dynamic or JavaScript-driven web pages which is one of it's major advantages.

from selenium import webdriver

# Configure the chrome webdriver
driver = webdriver.Chrome()

# Load the web page
url = "https://selenium.com"
driver.get(url)

# Extract data in the title tag using Selenium
title_element = driver.find_element_by_tag_name("title")
title = title_element.text
print("Page title:", title)

# Close the browser
driver.quit()

We can also combine Selenium with BeautifulSoup to get content rendered dynamically. Selenium automates web browser interaction. Hence the data rendered by JavaScript links can be made available with Selenium and then can be extracted using Beautiful Soup. Below is a snippet on how to use the combination of two.

soup = BeautifulSoup(driver.page_source, 'html.parser')
title = soup.title.text

Extruct

Python's extruct library is useful for extracting structured data from web pages. The extruct library comes in handy when you need to extract structured data from web pages, such as microdata or json-ld. It makes it simple to access and process structured information embedded in HTML. We need requests to load the web page data, just like BeautifulSoup.

import requests
from extruct.jsonld import JsonLdExtractor

# Make a request to the website
url = "https://extruct.com"
response = requests.get(url)

# Extract JSON-LD structured data from the HTML content
extractor = JsonLdExtractor()
data = extractor.extract(response.text)

Scrapy

Scrapy offers an integrated method for following links and extracting data from multiple pages. Scrapy is typically used to scrape data from multiple pages, follow links in web crawling, handle pagination, and perform more complex scraping tasks. It includes advanced features such as middleware, pipelines, and built-in asynchronous request handling. One disadvantage of Scrapy is that it does not support JavaScript by default, instead relying on Splash.

import scrapy

class MySpider(scrapy.Spider):
  name = "example_spider"
  start_urls = ["https://scrapy.com"]

  def parse(self, response):
    # Extract title tag data from the response
    title = response.css("title::text").get()
    print("Page title:", title)

Analyzing & Storing Data

Now that we have the data from the web, we can save it in the formats we want, such as CSV or databases. We can use Python libraries such as Pandas for data cleaning, transformation, and obtaining the final version of our preprocessed data.

Next, we can use Matplotlib or Seaborn to understand the trends, patterns, or correlations in the scraped data.

We can use Natural Language Processing to perform sentiment analysis on data containing customer reviews or movie reviews.

There are numerous applications for Machine Learning in scraped data.

Importance of Web Scraping

Web scraping is important in many industries. It aids in the monitoring of product prices, the analysis of customer reviews, and the tracking of competitors in e-commerce. Web scraping is used in finance for stock market analysis, tracking economic indicators, and collecting financial data. It is useful in investigative reporting and data journalism.

The applications are numerous, and web scraping enables businesses to remain competitive and make data-driven decisions.

Ethical Consideration & Best Practices

Some Ethical & Best Practices for web scraping include:

1. Respecting website policies: Check the website's terms of service and robots.txt file to ensure compliance with their guidelines.

2. Rate limiting: Implement delays between requests to avoid overwhelming the target website's server and potentially causing disruption.

3. Adding User Agents: Include a user agent in your HTTP requests that identifies your web scraping script. This allows website owners to contact you if needed.

4. Scraping public data: Focus on scraping publicly available data and avoid sensitive information or private areas of websites.

Hope this article helps you in getting a brief overview on Web Scraping & how it can be achieved using Python.

I couldn't help but wonder about the folks around me as I continued to examine them. What are their stories? What has brought them to this specific metro train at this specific time? What are their dreams and hopes?

Every time I am in a crowded place or stuck in traffic or traveling on public transport, I have a similar feeling where I can't stop thinking about how small I or my little concerns or issues are in the grand scheme of things, and how many different lives exist around me and the existence of infinite universes.

The sight of so many people in one place is both overpowering and fascinating. I cannot help but think about how many different experiences, emotions, and tales each person must have had. Each person has faced unique difficulties, joys, and struggles throughout their lives. Nonetheless, here we are, sharing a space and a moment.

Walking through a crowd of people reminds us that we are all connected and that, despite our differences, we all share a common humanity.

Taking a moment to notice and appreciate this difference is both humbling and enlightening.

One of the reasons I enjoy viewing the sky when walking is that it serves as a continual reminder of our place in the cosmos and the connectivity of all things.

Finally, busy areas or the sky are not only a voyage through diversity and quiet reflection for me, but they are also a reminder of the endless universes that exist around us and the huge assortment of sensations that exist.

1. Concept Drift & Data Drift: Concept Drift is basically when the relationship between the training variable & the target output changes whereas data drift is the change in the distribution of the data over time. Both can lead to a decline in the model's performance.

2. Software Engineering Issues: When we are deploying an ML Model there are certain factors that we need to think about, for example, the latency & the throughput needed, whether to have real-time or batch predictions. How to log the results for monitoring & maintaining the security & privacy of data.

When we train Machine Learning using a specific algorithm, the best way to deploy the model in production depends on a number of factors:

1. The acceptable downtime of our Machine Learning Solution.

2. The operation cost & the human involvement in deploying the model.

3. The ease with which we can roll back the model in case of a drift.

4. Whether there is a need to test with production traffic or not.

Now that we have understood what are the different challenges in deploying a Model, let's take a look into the different deployment strategies.

Recreate Deployment

1. In this strategy, we scale down the prior model before scaling up the new model version. Because it takes time to scale down the current model and scale up the new model version, the recreate technique is slow and causes downtime for the ML solution. Because we just have one version of the model, this strategy is incredibly straightforward to use. Recreate Deployment is not a scalable method and is best suited for small-scale applications.

2. We should use the Recreate Deployment when we can afford downtime with the product or when we don't want the new deployment to be backward compatible.

3. Example: In Machine Learning applications where we run the predictions in the form of batches.

Shadow Deployment

1. Shadow Deployment technique is used when we already have an ML model running in production. We used this technique to run the new model alongside the existing one in production. The forecast from the previous model is returned to the application, while the response data from the new model is saved for testing and comparing the outcomes. We require sufficient monitoring to access performance and must operate more servers for the new prediction service.

2. We should use Shadow Deployment when we want to test the new version across actual production data & at the same time don't disrupt the existing users.

3. Example: In Machine Learning applications where we want to forecast the business performance or growth, we can use shadow deployment to compare the predicted value from the model & actual growth.

Gradual Ramp-Up with Monitoring

The next 2 deployment strategies involve releasing the model for a certain % of users & then based on the performance monitoring, making it available to 100% of users.

Canary Deployment

1. In Canary Deployment, we have the old & new versions both running in production & serving the application. The major difference between canary & shadow deployment is that in Shadow, the response data from the new model is used for performance monitoring whereas here it is used to serve the application. The new model version is made available to a minimum set of users and then exposed to the entire set.

2. We should use Canary Deployment when we want to test the new version across actual production data & at the same time evaluate the existing user's response to the model. It allows us to spot problems early on before there are maybe overly large consequences to the application with no downtime.

3. Example: In Machine Learning applications that serve as recommendation systems like content or product. We can compare the interactions of different users with different models applied & then determine which was effective in providing recommendations.

A/B Testing Deployment

1. As the name suggests in A/B Testing Deployment, we have many different versions of the model. We divide the users into different groups based on the number of models we have & then decide the best model based on the performance & the user's interaction. With A/B testing we can discard the low-performing models fast with no downtime.

2. We can use A/B testing deployment when we have a couple of models which provide almost similar results. With this technique, we can determine the best model using production data & response.

3. Example: Similar to Canary Deployment, A/B Testing can also be used for recommendation systems like content or product recommendation.

Blue-Green Deployment

1. The blue-green deployment is accomplished by utilizing an existing prediction service. Then, as the staging environment, we build a new prediction service, the green version. Once the performance and functionality testing in the green environment is completed, we have the router switch traffic from the old to the new. It incurs additional costs due to the upkeep of various settings. The benefit of a blue-green deployment is that it enables simple rollback. If something goes wrong, we may simply reset the router or switch to the blue version to divert traffic.

2. We can use Blue-Green deployment when the application can afford no downtime and backward compatibility is required.

3. Example: Real-time prediction system like fraud/anomaly detection.

Below is a brief table of differences based on the explanation of each method and the four main considerations we mentioned above for picking a deployment strategy. I hope this article helps you select the right deployment strategy for your Machine Learning applications.

As a contestant, I believed it was simple for the judges to determine the outcomes or how simple it was for them to give the teams advice. Being on the other side of the table helped me to realize that both sides require the same amount of preparation.

There is a similar sense of anxiety or nervousness on both sides. Developing and pitching your solution properly is the biggest challenge for a contestant while judging your solution based on a fair evaluation is the biggest challenge for a judge.

My primary learnings from the experiences were the following.

• The contestants must design a distinctive solution to the problem, while the judges must understand the problem statement, think of all possible solutions, and evaluate using the right technology.

• Similarly to how participants need to be prepared to deal with any questions that might come their way during the pitch, mentors need to have answers/suggestions to questions participants might ask.

• I realized during the mentoring session that you need to control the developer inside of you. When teams came up and mention bugs/errors they are experiencing, it's tempting to grab the laptop and fix them. However, you need to control yourself and provide them with the right information.

• Just as teamwork is required in participation, it is also needed between the judging panel members. An agreement needs to be reached on the evaluation criteria and the scoring system.

Thus there's no need to be intimidated by the evaluator if you're just starting to take part in hackathons. Keep in mind that both parties are under equal pressure, for the participant it is to produce the finest idea and pitch, while the judge is under pressure to render a fair judgment.

Being my first time doing one, the prospect of presenting a Tech Talk was both exciting and scary.

Now that I had to give a talk, the most important thing was to find a conference. Since I had been working on Ruby on Rails, the idea was to find a Ruby-related conference and submit the CFP there.

In November, I came across the Ruby on Rails Global Summit by Geekle. I intended to attend this summit, observe the speakers' presentations, and then submit a CFP for any upcoming Ruby conferences I could find. The CFPs were still open though.

I told my idea to Hari, he convinced me that since the CFPs are open, I should submit one & if I don't get selected I can always come back to attend the summit. This way my plan of attending the summit was now changed to speaking at the summit.

Submitting the CFP

Since the CFPs had been open for some time, many of the topics I was thinking to speak on were already taken. I finally decided to speak on Rubocop, since I had been working on it for some time. The plan was to show a live demo on modifying cops & showing the before & after results. After various iterations on the abstract & title I finally submitted my first ever CFP - Enhancing Code Quality with Rubocop.

During mid-December, Liliya from the Geekle team reached out & yes the CFP was accepted, I was now giving my very first Tech Talk.

Preparing for the Talk

Since the talk was in January, I was pretty chilled out when I received the acceptance. I had already written a blog on Rubocop so I felt that I could manage the talk.

When I got another email with information on the platform we would be utilizing and promotional banners for my talk, that's when I started to freak out as I realized the scale of the conference. To make sure everything was set up, I was requested to schedule a test call with the technical team.

During the technical call, they walked me through the Streamyard platform, how screen sharing works, and all other minor details.

When the call was over, that's when I first felt tense. I started overthinking even simple things like screen sharing because I was so new to the platform. What if I share the wrong screen or my internet goes out? Furthermore, I had to code live in front of my audience. What happens if my code malfunctions? I began to wonder if I ought to have recorded the presentation beforehand and submitted it. But it was time to push past your comfort level.

To avoid any problems that might arise during the discussion, the idea was to prepare the slides and run through the full flow as many times as I could. Also to get an idea of how people usually do it, I went to youtube & watched various talks given by developers I had heard of.

Practicing the talk

Once I had the talk ready, I practiced them & that is when I realized, I need to restructure the slides. Since I had a combination of theory & code, I had to make sure there was proper spacing between the two, so that the content was easy to consume. I reordered and practiced again. I realized the speed at which I was talking was too fast & need to slow down.

I showed Hari a demo of my talk two days before the event, and he offered me revisions as feedback. This was quite helpful. The most significant piece of advice I received was to interact with the audience & make the talk more engaging.

I began to consider ways to include audience participation in the presentation. The options for participation were restricted because it was an online conference and nobody's video would be turned on. As a result, I incorporated surveys in the form of QR codes and invited the audience to participate in the coding.

Once I had the final slides, I did a final demo talk with Ishan & again with Hari. Giving demo's helped in boosting my confidence.

On the day of the talk, I played around with the commands & the code two or three times to make sure that everything was running as expected.

Going Live

All the nervousness vanished as soon as I began to speak because I became absorbed. During the talk, I wasn't able to see whether people were answering my questions or the poll but when I completed the talk & checked, I was happy to see that the audience engaged with me & answered the polls. Some people even reached out on LinkedIn with positive comments.

Overall, it was a fantastic experience & I am glad I went live instead of pre-recording.

PS: You can find my talk here :) https://youtu.be/H7ggpT_ZFIQ

What is a retrospective, to begin with? Wikipedia's description

A retrospective (from Latin retrospectare, "look back"), generally, is a look back at events that took place, or works that were produced, in the past.

First, why is it important to write a personal retrospective

• Avoid Recency Bias: It is a typical fact that we tend to remember recent events better than events that occurred in the past, giving the recent event more significance. It's wise to pause, take a step back, and reflect on our past actions in order to avoid this bias.

• To comprehend what you have learned: To identify the important areas where you need to improve, make a list of all the lessons you've learned.

• To consider what went wrong: Reflecting on situations when your work fell short of expectations or where you encountered difficulties will help you identify potential mistakes you shouldn't make.

• Understanding your preferences: There will be many different jobs that you have worked on. While some may be of interest to you, others won't. You can request more tasks if you have a detailed list of your preferences.

What can be the format of the retrospective?

• If you worked on several projects over the quarter or year, you can break up the retrospective into each project's learnings and successes part.

• The alternative strategy involves combining ideas like learning, successes, failures, preferences, and dislikes.

How can you start with writing your retrospective?

• Gather all the data points: This, in my opinion, is the most crucial action. You can review the data from the SCM you use to understand all languages, code bases, and even the kind of issues you worked on. To learn the theme and the list of languages, look at the commit messages and file types. Moreover, you can look at project management systems like JIRA to determine whether you focused more on Debugging or Features throughout your time. Review all the different architecture-related guilds that you have been a part of.

• Random notes from the past: Due to recency bias, which we discussed above, there are likely to be some little but significant events from the year that you will soon forget. These may be difficult problems to fix, a team discussion about the architecture, a few support tickets, or anything else. The notes don't have to be flawlessly formatted; a few bullet points can do the trick and serve as a recall of the entire scenario when you go back to it. I personally record everything that comes to me using the self-messaging function in Slack.

• Initial Thoughts: Wait a while before beginning to write a well-organized reflection. Make a list of all the things in the form of bullet points that come to mind when you think of retrospect as you stop. I usually prepare a list of every point I want to make in my retrospective before going into more detail on each one.

  Even while we may believe that writing a retrospective is simple, it actually requires a lot of thought and perseverance.

  A personal retrospective is written similarly to a Root Cause Analysis of any technical incident, with the exception that in this case, the RCA is based on the individual.

A debugging process can be divided into two categories:

• Active: In this step, you investigate all methods of fixing the bug by altering the code. The next steps are developing a hypothesis as to whether the solution will work and implementing it.

• Passive: For passive debugging, we use logging when the bug cannot be reproduced, so we cannot run through the execution steps.

Throughout this blog post, I'll share some of the debugging practices I adopted and have now become part of my debugging process.

It is essential to remember that if a bug exists, even if you are not experiencing it, it implies that something is definitely off with the code. This could be logic, a dependency issue, or a missed edge case.

Go Back in the History

Gathering all available historical information about the issue is the first step in debugging. This typically allows you to identify whether the problem is persistent or whether it has never been fixed. There may be a log, error note, screenshot, and perhaps some old message exchanges.

SCM also allows you to view all of the file's previous modifications. Different Git extensions in the IDE can help with this process. Going back in time and discovering even the smallest indication might occasionally inspire an idea when you are absolutely baffled as to how the bug is happening.

Read Read Read

Sometimes, half of the issues may be solved by just reading the code and tracing back all the functions.

In my experience, after locating the API Endpoint where the problem occurred, I look for the method connected to it and then attempt to visualize the flow by creating a mental model of the code.

Once you comprehend the code, you may add breakpoints (eg: binding.pry in Rails) to the code to verify your comprehension.

Talk to Yourself

When working on a complicated problem, speaking aloud to yourself might occasionally disclose blind spots.

I try putting it down in the manner of what I would advise someone to do if they were in a similar circumstance if thinking aloud doesn't help.

Simply write down everything you know about the bug so that when you see it all written out in front of you, your mind will start to fill in the blanks.

Add Logs Everywhere

One of the most popular debugging techniques is this one. It entails inserting logs into the code at various locations—basically, before and after every function call.

Adding logs can help you understand the state of a variable whose value is changing across many functions.

Play Around to Reproduce the Bug

As a result of the discovery of the defect, we can conclude that the code is flawed. In huge code bases, it is essential to reproduce problems that only appear under a certain set of circumstances.

In order to understand the set of conditions, all the data points can be gathered as you analyze the database values. Try finding a pattern across all the users for whom the error has been occurring. You can then experiment with a different set of conditions to correct the issue.

To further aid in the debugging of edge cases, the code may be significantly modified to include early returns, conditional returns, etc.

Forgetting what you know about the product is a crucial step in trying to recreate something. Typically, when we have been working on something for a very long time, we are familiar with all the tricks and know what will work and what won't. Due to this, the bug's reproduction is biassed. Consider your product from the user's perspective and create the most absurd scenario possible.

It's not always some Big Change

When you encounter a bug, don't always think of the issue to be some big code change or rewrite. Start small.

Check for null values, any before action validations, if else conditions.

When I originally started, I believed that significant modifications would be necessary, but over time, I've learned that it's better to focus on the little aspects first.

Read the Unit Tests

You can start by reviewing the unit tests for that specific piece of code if you are entirely perplexed by the various edge situations the function handles.

Talk to the Code Owner

There is a possibility that you will not always have to debug something that you have worked on. When this happens, it can be helpful to speak to someone with a broader understanding.

You can bounce your ideas for solving when working on a critical flow, if modifying something, with the working team to ensure you haven't missed any edge cases.

Have a Proper Alert & Monitoring System

Using tools like HoneyBadger, and Sentry can help you keep a log of the errors. You can add them to all your APIs & critical workflows.

Other than alerting, you can have a logging system, where you can monitor the routes that are being called in actual production.

Document it

The most crucial step is the last: once you've identified the cause of the problem, remember to record it for the future you.

Lastly, if none of the above steps work, take a break, come back and start again.

PS: Keep in mind that debugging is not magic; rather, it is a sequential process that you get better at as you do it more frequently.

Document Everything

Documenting can be classified into four types:

• The tasks you need to complete: Teams use product management tools to track tasks, but maintaining a personal TODO list is extremely helpful. On a regular basis, the list can be created. In retrospect, you can mark off the tasks that you were able to complete. I usually like to maintain a daily TODO list.

• Maintaining your blog: Recently, I also began this & it has helped me organize my thoughts and easily document the things I have learned. It helps you think about your problems and communicate them more effectively when you write.

• The work you have done: Consider keeping a journal of your work so that you can reflect on your performance. As a result, you'll have a clearer idea of what you've learned.

• Things that pop up randomly: While working, you may get some questions or find something interesting related to your work, or the approach to solving a problem may be complicated. Be sure to document it. The complicated approach becomes easier to convey when you write it for yourself.

If something doesn't work, leave it for sometime

In development, we often end up coding all the time while trying to solve a problem/bug. The brain reaches a saturation point at which it is no longer productive.

When I would get some issues in the past, I had a policy of not sleeping or taking a break until the bug was resolved. It only took me some time to realize that having some breaks reduced the time it needed to fix the bug.

Now when I get stuck on code, I take a break for 10/15 minutes, and when I returned, the code ran fine with minor adjustments.

Sleeping over an issue has also been beneficial. As soon as I awoke after sleeping, the solution that I had been unable to find after hours of research came to me in minutes. Sometimes the solution felt like a Deja Vu dream.

Not always about coding

Being a Software Developer doesn't mean you need to code daily.

It is sometimes necessary to spend hours reading the code to figure out the root cause of the bug, gain a better understanding of the issue, or find different approaches to the problem.

There have been some bugs where the actual code change was only 1-2 lines, but it took me hours to research and go through the code.

Consistency

You don't have to be an expert at everything all the time. You will become proficient at it only if you practice it repeatedly.

The first time you do something similar, you will probably make mistakes. However, the next time you do it, you will already know what to do and what not to do. Your brain will follow along automatically.

You Never Stop Learning

Technology is a vast field that is advancing every day. The work you do on a daily basis will teach you things, but it usually happens in a chaotic way.

A book, course, or blog can be very helpful for learning stuff in a structured manner.

Furthermore, you can gain knowledge from other developers with different experiences & journeys. Learning is a continuous process.

Every piece of code should be run once

Whenever I make a change to code, I test it after someone reviews it and asks for a change, and before merging it to master after getting the approval.

You never know what might fail if you don't test every change. You're better off delaying a feature than pushing one that has flaws.

Whenever you write code, you should test it before you put it into production.

The process of breaking down into smaller pieces

Take each part of a problem and divide it into small achievable goals, whenever you encounter them.

My usual approach is to think about the dependencies involved in solving the problem. Then search for solutions to each of the dependencies. Your brain automatically gets motivated to tackle the larger problems as soon as you begin solving the smaller ones.

By creating a flowchart or list of smaller tasks to be completed, you have the requirements clearly in front of you. All you need to do now is think of the solution.

These are some of the things I have learned so far as a Software Developer. I hope you find these tips useful in your journey as a software developer at whatever level you find yourself.