ARV's Blog

July Journal

Anshul Raj Verma — Wed, 01 Jul 2026 00:00:00 GMT

Week 27 Journal

Amazing tool datamodel-code-generator to convert OpenAPI spec into Pydantic Models. I am using this in the Zyou SDK.
Working heavily on Validator API <> SDK <> MCP.
Took one day off due to headache, but also did some work after feeling well. It maybe happened because late night films and shows watching.
Played Badminton with Team.

New open specification introduced for AI Agents, currently in early stages but looks promising:

Open Knowledge Format (OKF):

1. `index.md` will get generated by a script not by the LLM.
2. A MCP server with tools to query the OKF Bundle.
3. Maintain a JSON schema to help the MCP server/tools.
4. Concept links validator to validate whether the written links exists or not.
5. Also create MCP tools for `index.md` and `log.md`.
6. MCP tool for accessing `# Citations` from given file path.
7. An UI for the OKF Bundle for better readability for Humans. **/w Astro**

Agentic Resource Discovery (ARD): It defines how agents and tools are cataloged, indexed, and searched across federated registries, so an agent can find capabilities at runtime instead of needing them preinstalled. It is a shared standard that any company can implement independently, and that any agent can participate in.

Installed OSS Handy app for Speech-to-Text work.
I registered okf, arv-anshul, and anshulrv package name on pypi.org. arv and anshul are already registered.

June Journal

Anshul Raj Verma — Mon, 01 Jun 2026 00:00:00 GMT

Week 23 Journal

PAXEL by Y Combinator uses a shell script to analyze all of your session with Coding Agents like Claude Code, Codex CLI, and Cursor to generate a report and a profile based on your prompts, activities, and tool calls. You have to run a command in your terminal to generate this report. YC doesn't stores anything regarding your sessions and all. At the end, a JSON payload of scores, narratives, redacted decisions, and session metadata is uploaded to YC.
Thought:

Ab aapka apna koi nahi hai,
Ab bas aapka apna ye hain aur aapka apna ye hain,
baki sab matlabi,
Yahan tak ki mai bhi jo ye kar raha hun shayad,
ye bhi main apne liye he kar raha hun,
ki kal jab aap na raho to main ye soch sakun,
ki jab aap the tab maine ye kiya tha aapke liye.

Week 24 Journal

@Pratham came from Home. Now, we started working on SDKs and MCPs. We are now watching "From" and already completed the Season 1.
Me, @Anurag and @Pratham watched Obsession movie in cinema hall. It was horrifying.
Choice filling of Colleges for @Akansha on JOSSA and COMEDK.
Wrote some helper Agent Skills.
@Pratham is switching to Neovim, it's fantastic. I was also thinking to switch to Neovim or Helix but decided not to because I've a grip on Zed IDE already switching from it will be difficult and I'll some features. But I'll definitely learn Vim key bindings.

Week 25 Journal

Gone for TC and Pool Party with Friends and Co. to Bhopal.
@Akansha is learning video editing and some camerawork for content creation.
Watched the whole season 3 in train of "From".

Week 26 Journal

Completed "From" season 4, finale was mid. Now waiting for season 5.
I worked on my Claude Code configurations to ease my coding workflow:
- RTK (Rust Token Killer) CLI hook which reduces output tokens of widely used CLI's like ls, find, git, etc.
- Mattpocok Skills comes with many useful skills for developers my favorite /grill-me.
- CLAUDE_LSP_TOOL pyright-lsp for Python. But Claude doesn't use it often time.
- /ponytail Agent Skill works nice with Opus 4.8.
- Disabled Grep and Glob builtin tools for rtk grep and rtk find CLI.
- ccstatusline to show important session and context metrics at the bottom.
Apple increased it’s prices.
Pull out two all nighter with @Pratham and Team to work on projects.
Gone for team dinner at MomoBae.

AI Harness

Anshul Raj Verma — Sat, 30 May 2026 00:00:00 GMT

I have started using Claude Code recently in my projects but I am find it hard to maintain its configurations because Claude Code has it own set of convention than other AI Harness. Due to that I have to maintain both .agents/ and .claude/ which doesn't looks good. Why do I maintain the same thing at two places (even a symlink).

I have switched to chezmoi just because of this.

Although, Claude Code is known to be as the best in its class but I want a better more acceptable, flexible solution to this. And there is two in the market OpenCode and Pi Agent. OpenCode is more popular than Pi but Pi is more flexible in nature. I am choosing Pi.

Even though I will not customize it a lot. It just me who wants to try and stick with Pi.

Requirements

A coding agent should have these things:

Community Convention Follower
Agent Skills
AGENTS.md
Plan Mode
Open Source
Models Availability (No Vendor Locking)
Extensibility and Customization

Recently I have Vertex AI project access through which I can use any models like Gemini, Claude, GPT, etc.

Agent Skills

The coding agent should support the Agent Skills. This is a standardized way to give AI agents new capabilities and expertise.

AGENTS.md

Think of AGENTS.md as a README for agents: a dedicated, predictable place to provide the context and instructions to help AI coding agents work on your project.

Many agents follow their own conventions like Claude Code has CLAUDE.md, Gemini has GEMINI.md (it also supports AGENTS.md) but I want an agent which supports this AGENTS.md convention because it is being decided by the community.

Plan Mode

I don't know how does Plan Mode works in Claude Code but it is very useful for me to think about a feature implementation before the actual implementation. I think there is one simple solution to use an Agent Skill for that but I am also exploring more in community.

I have found @ifi/pi-plan (which is actually inspired from /plan-md) and I think this is a better tool for planning task because it has a workflow pipeline which guarantee idempotency over tasks.

Open Source

There are multiple open source coding agents like Codex CLI, Gemini CLI, OpenCode, Pi Coding Agent, etc. But Codex CLI and Gemini CLI are vendor locked means you cannot use other provider's model in those CLIs.

OpenCode is the most popular one here but I want to go with Pi Coding Agent due to its extensibility and customization.

Models Availability

Claude Code, Codex, Gemini are vendor locked but OpenCode and Pi are not. You can use any provider's models in those coding agents which is very useful.

Extensibility and Customization

Pi is very minimal coding agent but you can also customize it with extensions, prompts, skills, and themes. See Pi.dev

I don't need shiny and bloated features in my workflow. I will keep Pi minimal with aesthetic yet powerful and performant agent for my workflow. I will maintain a repository where I keep my customization configurations.

May Journal

Anshul Raj Verma — Fri, 01 May 2026 00:00:00 GMT

Week 19 Journal

Busy in office work.
I haven't done meaningful other than my office work which I cannot mention here that's why I am able to write it consistently. Now I use a dairy and pen to make notes while working.
Sometimes I write couple of paragraphs on Google Keep which I've decided to put them here instead.

Week 20 Journal

Busy in office work.
Shifted to new flat. Busy in arrangements. Eating from restaurants for 2 weeks.

Week 21 Journal

Busy in office work.
Visited cousin's house on Sunday and came back on Monday for work. I struggle with the buses even now.

Week 22 Journal

Busy in office work.
@Akansha got her new Macbook Air M5.
Thought:

Aapne agar dhyan diya hoga to jante hoge ki mai jyada nahi bolta, jyada jhagadta nahi.

Pata hai, kyu?

Kyuki main aaj aapke paas hu, apse aankh se aankh mila kar baatein kar raha hun.
Lekin kal jab main akela rahunga aur jab aap mere yaadon me aaoge tab main ye nahi chahta,
ki unme sirf meri baatein, aur aapsi jhagde mile.

Mai un yaadon me aapko dekhna chahta hun, aapko sunna chahta hun, aapki baaton pe hasna chahta hun.

After writing this, I read my Google Keep notes and found one para which intrigued me and I started questioning that it actually written by me or I copied but it is written by me. Although, I have not remembered why and how I wrote this but this is a good one (my POV).

Man ko janjhol raha hun, tatol raha hun,
Sabdoon ki baadh bhi aayi hai,
Ab toh bas sabdoon ki mala he pironi reh gayi hai,
Lekin saamne rakhi ees nacheez ne rok rakhi hai,
Jab dhaara me chhlang lagane jata hun,
Ye mujhe kheech lati hai.

Goa Trip

Anshul Raj Verma — Tue, 28 Apr 2026 00:00:00 GMT

24th April

With Pratham and Anurag sir, we moved around 09:15 from Pune guest house. We took the 100km extra long route to see the scenery of Mulshi Lake. We had Sandwich and Cold Coffee at a restaurant nearby. We reached our rented flat at ~9 o'clock. We checked-in and immediately walked to Fish Deck a nearby restaurant for dinner. After that, we decided to see the beach which is around 1.5km away. Just after 400m-500m of walking we felt the humidity and sweating it was disorienting of will to go there but eventually reached and enjoyed the scenery of Anjuna Beach. I called my college friends in Bhopal who doesn't know that I was on a trip to Goa. That is intentional because I wanted to surprise them and flex before them on the group video call. After sitting for some time on the beach, we returned to our flat to bath and slept.

25th April

Next day, I woke at 12 noon. Anurag sir has gone for a haircut and we go to Anjuna Flea Market to buy some clothes which is suitable for this humid weather. We haven't ate that day until 17 but eventually found a Punjabi restaurant where I ate Chicken Biryani and Lassi, it was fabulous. Me and Anurag sir ordered the same Hydrabadi Chicken Biryani each, when the order arrived it was huge we should have ordered only one for both of us. Even then I ate all of my part but Anurag sir left some of it. Pratham sir is vegetarian so he ate Paneer, Roti and Lassi. Then we go to Baga Beach which is full of crowd. We stayed there about 10-15 minutes and moved to pick up Aditya sir from Dabolim Airport. Finally, we shake hands. With Aditya sir onboard we reached our flat to get ready for the Savara Night Club opening. Manik also came to the flat to pick us.

It's my first time at a club. I checked one point in my experiences list after experiencing that I had written some words for that:

I think, it increased the hearing and lower my thoughts handling capabilities. I am feeling a little dizzy not too much, its just I am moving faster. It's like everything seems faster now. Surprisingly, I feel confident and I am not able to concentrate on my thought for a thought I just do whatever comes first. I gone to washroom and choose the very first seat and this is seriously not me. This feeling faded away after sometime (I was keeping track of time).

While having it I was just thinking one thing, why someone wants this bitter taste constantly to feel unconscious, there are better alternative.

Around 3AM we reached flat and slept.

26th April

We had lunch at Goan Spices and came to Anjuna Beach to enjoy the beach and sea. Only Me, Pratham and Aditya sir bath in the sea, Anurag sir stayed at shore and guard our clothes and phone. We didn't rest in the sea. We stopped the waves so that nobody behind us will be able to play with it. After some time I found a ball floating in the sea and then we started catching and diving with the ball. Another group of boys also joined us in the play. We played it to the fullest. Then we had dinner at Fish Deck.

27th April

We had a meeting, so we gone to Coffee Central cafe for that. After that we were wandering around in Flea Market and around 21 we left.

28th April

Reached the guesthouse around 9.

How to explain a face from your thoughts?

Anshul Raj Verma — Thu, 02 Apr 2026 01:19:00 GMT

Suddenly at evening (01 April) while enjoying the rain with @Ankit and @Kunal on the hostel's roof. I stuck on a face in my mind whom I cannot define to others. I can't recall his identity. I am certain that he is real and he is a teacher or similar because he is explaining something and giving a smile.

Even now while typing these I don't know who he is, where I have seen him but I am stuck at him. Whenever my mind is empty, his face comes up. He is smiling. He is speaking something. These expressions of his of 1-2 seconds, keeps replaying in my mind like a GIF.

I think I've seen him in college or on YouTube, I'm not sure. His face is very clear in my mind but I can't explain it.

Right now, I need those sketch artists from crime movies who draw the faces from explanation.

April Journal

Anshul Raj Verma — Wed, 01 Apr 2026 00:00:00 GMT

Week 14 Journal

Watched movie Edge Of Tomorrow.
Watched a Netflix show American Vandal, it was amazing, too good. It has been on my watchlist since 2023-01-17, as recommended by ComicVerse YT channel.
I also watched couple of episodes watched The Mentalist but it is boring. It isn't gripping and thrilling as I expect to be.
Spotify updated their policy for Developer Access, see blog. Now I can't be able to maintain my Spotify data through arv-anshul/spotyhive repository.
IDEA: Create a route (because currently it is static) in the website which can serve a bash file to render the website gist in the terminal. Similar to curl ysap.sh.
IDEA: Create a CLI app which can render the resume.json as a Resume in terminal.
IDEA: Create a program which creates a proper structured notes from YouTube video.
Had a talk with @Adityaojas and @Manik for the migration. I might move next week.
Connected with @Araadhay for his new idea.

Week 15 Journal

Helped CampusX to protect their notes from leaking. They had showed their notes URL in thier video through which anyone can download it. It was stored in Google Cloud.
Watched Shrinking Season 3 and an anime Dohedoro.
Had a meeting with @Araadhay regarding his new OpenClaw and ClawHub related idea.
Hermes-Agent a lightweight OpenClaw.
Confirmed the date of migration from Hostel to Pune with @Manik, it is 15th March.
@Naina wrote a formal invitation for her birthday. It was quite unexpected but I really liked it.
Had a long day on Saturday because of Scooty servicing (5 hours), gift decision (4 hours), bath.
Enjoyed at pool party on @Naina birthday party.

Week 16 Journal

Came to Baner, Pune on 16th at 9 o'clock with 16 hours late train.
Met with @Manik, @Pratham, and @Anurag at the Guesthouse.
Visited cousin's house on Sunday and came back on Monday for work.

Week 17 Journal

Working from 13 to 21 in office.
Taking a trip to Goa from Pune.

Week 18 Journal

Enjoyed Goa from 24th April to 27th April. See the blog on it, Trip to Goa
Team had a meeting in Goa.
We shifted to our flat.
From now, I will write lesser because for the most of the time I hang around the team and it is not good to journal them publicly. Instead, I will try to write down my thoughts and experiences in diary.

96 by C. Prem Kumar and Govind Vasantha

Anshul Raj Verma — Tue, 24 Mar 2026 00:00:00 GMT

These days I am constantly in doubts, not able to think clearly, not able to work on anything. Not able to think anything. Don't know how my day goes. Just linger around social media (YouTube, GitHub, Folo). Not enjoying anything music or movie, nothing.

Then I decided to watch the movie "96" because related content is coming as recommendation on the YouTube home page.

I think this is a slice-of-life romantic drama film. Although I've never get into these emotion deeply, but I've scratched it alone a long time ago. I have been suppressing this emotions for about 5-7 years because I want to understand it first but it is hard in the current surrounding and environment. So I seek these films and shows like 96, Past Lives, Meiyazhagan, Dead Poet Society, Gulmohar, etc.

मेरे लिए वो मूवी प्यारी है जो मेरी आँखों में आँसू ले आए। लेकिन यह मूवी मेरे लिए उससे भी बढ़ के है।

I have discovered the soulful music of Govind Vasantha and this week (Week 13) I am constantly listening to his creation. He even composed the music of Meiyazhagan; isn't it awesome, that loved it without knowing that it is composed by Govind Vasantha. I would like to thank him to compose these soulful music. I've been listening him this whole week and always will.

For me, the role of BGM, OST and integrated songs in movies and shows is too important because I've discovered some of my favorites from movies and shows I had watched. I have created multiple playlists and saved albums from those. I also listen pay attention on them while watching and try to navigate, what are they conveying.

Dehradun Trip

Anshul Raj Verma — Wed, 18 Mar 2026 00:00:00 GMT

Its again only us, Anshul, Ankit and Tanu Didi 6 months after Jaipur trip. Dedicating our next 4-5 days to Dehradun Trip.

There's so much happened in between these 6 months.

I joined my college, made several awesome and fantastic friends.
Stuck my life between college practicals and assignments.
Traveled 3 times to home without reservation in train (it should also be called as an experience).
Took one of the important and hardest decision to drop out of college, so that I can go solo.

I Shouldn't Go

Trip planning is the responsibility of Ankit and Didi, I just review it and give feedback on it. While the trip is at its planning stage I was indulged in the process of TC and Ankit is in his semester exams.

I was having a hard time on how to get along in life now and there is no case to go on a trip. I want to reflect upon myself alone while Ankit will be on the trip. But it didn't happen because I am not able to communicate my thoughts.

Also I have decided to go to next trip with my sister and that is not happening.

In the Trip

Almost 2 hours before the train to NZM, I was convinced to go on the trip. I packed my couple of clothes and put my shoes and got ready for the trip because it was the time to not get bored by enjoyment. I forgot the past and lived in the moment. I sacrificed my past and future to live in the moment.

Again I don't have reservation but I bought one general ticket but when the TT checked it isn't the valid general ticket as the train was superfast and I've taken Mail/Express ticket. The TT told to get the penalty ticket but when it was my time he was distracted by another and forgot to charge me. Again, I got lucky. (I know this isn't good but there's no choice)

Day 01: Awesome Chhota

From NZM, we are going to Ankit's uncle's place to stay till 21:00 so that we can catch the 23:00 bus to Dehradun from Anand Nagar ISBT.

I met an awesome, artistic, creative girl named SHNY and her older sister PRNM. I specially want to talk about SHNY. She looks very similar to one of Didi's friend.

Approx 2 hour after we reached, she brought some paper made stuffs like bag, birthday diary, spray, laptop, and many things. At first thought she would have made these from YouTube videos but I was totally wrong and it's amazing that a girl this small can create these crafts on her own just by looking and observing things :salute:.

I was astonished by her Laptop, Birthday Diary, Spary Can and I want to demystify each of them and you will also tend to appreciate her creativity.

Birthday Dairy

It consists of four pages each saying a sequence of story.

Cover page says "Happy birthday to you".
A cutout of cake depicting the celebration.
A cutout of flower to bring color and happiness in the life.
Last page stating the smile on the face of the birthday girl.

Laptop

Her sister helped her to create this but the most important and nuanced part of this is creation is built by SHNY. The keypad of the Laptop where anyone of us would just thought to draw the keys but she is different and a nice observer. She created a spring mechanics by folding the paper for the keys and this detailing is fantastic. The laptop also has icons of files, folders, YouTube, Settings, etc.

Spray Can

Once her cousins were playing with some spray and they are not letting her play with it and that's when she decided to built it her own. She built it with proper detailing like the Can Cap, Tip, Spray foam.

She is just awesome and I wish she would do or create something even more awesome things.

While leaving, she cried out loud but there's nothing we can do but leave.

Day 02: New City, New Experience

Its time to explore the city from ground by searching for a good hotel with reasonable price.

I thought we can find it with the help of Google Maps but it is quite difficult without any guidance and wheels. A auto driver approached to show some 2-4 hotel with reasonable price in around ₹50. We accepted the offer and after 4 hotels we reached a reasonably good hotel with affordable price.

Hotel: Aryan Castle Homestay, Shimla Bypass, Dehradun

We took the single bedroom at ₹2k for one day stay around 07:30. Gave ₹100 to Saheed Ahmed the auto driver.

After changing and resting in about 2 hours we came out to search the bike rental shop to rent 1 bike for three of us which is quite risky. We asked Saheed Bhaiya (Auto Driver) and suggested "UK07 Bike Rental Planet" but at that time it was closed. So we searched for another on Google Maps, it was 2.5km far, took auto at ₹80 but couldn't find a right bike so we fallback to previous bike rental.

We took "Apache RTR 200cc" for ₹900/day and "Royal Enfield Hunter" for ₹1300/day.

After a nice list of traveling point from Bike Rental wale Bhaiya, we first moved towards FRI college. It is well known for its Bollywood film shooting place. After reaching there we came to know that it was closed for tourists for almost 6 months. Without wasting much time we moved towards Tapakeshwar Mandir, we just have to face traffic for couple of minutes. After pray before Lord Shiva and other Gods, it was the first time we touched flowing water on the trip. It was so cold and mesmerizing. It was a moment. Although we (everyone there) are busy taking pictures and videos but no-one not even me tool a moment to live the moment or try to thank the calmness of the flowing water.

I believe, traveling is not capturing the moment instead it is a way to live in the moment by forgetting each and every of your thoughts and problems.

It is hard to follow up on this thought because I had already tried it on Jaipur Trip I couldn't hold it even there. So I just go with the crowd and did what others are doing.

After playing enough with slow stream, we ate a little in nearby Dhaaba and moved towards Robber's Cave (Gucchu Paani). It also has stream but it was speedy and longer. Also it has small waterfall in the end after 300 meter of walk but we couldn't bath because we didn't plan it and the water was too cold but we walked till the end, it was awesome.

After enjoying there, we are planning to meet on of Didi's friend who study in Dehradun. But first we stopped by on roadside to take some cinematic and photos. In the middle of the shot heavy rain and thunder came due to that we stopped at nearby bus stop shade. Just after 5-7 minutes to standing, a guy on scooty slipped on the road because he was speedy and couldn't control himself on the slippery road. Its good that he was alone on it and with helmet. I think he got a small injury in his leg. People at the bus stop bring him in the shade and told him to rest till rain stop but after 5-8 minutes he's gone with his injured leg.

As the rain slowed down, we also moved and reached hotel fully soaked. The rainwater was traveling from head to underwear, even the shoes got wet and the cold weather was cherry on the cake. We changed clothes and took some rest. It's 7:30, rain has stopped, we are hungry, and Didi's friend is also up for meetup so we decided to ride again. But there was a problem I didn't bring spare shoes as my shoes was soaked, so I wore the bathroom slipper given by the hotel. If you are worried about Ankit and Didi then do not because they brought three pairs each.

Didi's friend was 12KM away from hotel, the weather so cold but it wasn't raining. We eventually reached and we all sat for dinner. At the dinner table we discussed whether should we ride to Mussoorie because of this rain I was worried so much and at the same time rain started again outside. After finishing the dinner the rain stopped, so we quickly moved towards hotel. We are also searching for shops to buy warm clothes for next day. I was feeling even colder after dinner on the bike with the slippers.

We are going to check out from current hotel and take new in Mussoorie to stay at night there but we have one suitcase which can be quietly cumbersome in the ride. We decided to drop the suitcase at Didi's friend's hostel although it might cost us 24KM more but it was better to leave the suitcase behind.

Day 03: Ride up to Mussoorie

We woke up at 08 AM, everyone is getting ready one-by-one we planned to check out by 10 AM. Before getting fresh I gone to check upon the bikes, they look good. While returning the owner of the hotel called me asked when did you checked in yesterday? I didn't know. He checked the register and told you did at 07:34 AM and now it is around 08:10 AM which is more than 23 hours. I was confused because yesterday receptionist told that we can check out by 11:00 AM. But the owner explained and accepted his mistake and asked for penalty of ₹300 and then we can check out by 10 AM. I got up in the room and explained the situation. We thought don't get involved in this, just pay and move.

We learned a new experience that before check in asked about any penalty and the time of check out, clearly.

We changed the plan to drop the suitcase to Didi's friend's hostel instead we are going to drop it at Bike Rental Shop which is 800 meters away from our hotel. Now we gear up the Map to Mussoorie on phone and triumph the bikes towards Mussoorie.

Took multiple stop while climbing and took awesome pictures. Stopped at a Lord Shiva Temple. We also stopped at a cafe (Green Valley Cafe) on the roadside for Maggie and Momos and a nice look of view which have climb. After 1.5 hours of ride we reached the top and now we are looking for Hotel. Just where we stopped a woman approached us and suggested a hotel to look. We thought start from this if we not like will move to other.

We parked the bikes just below the hotel, yes below because you have climb almost 50-60 steps to reach the hotel. They showed two rooms one has no view and one has very nice view. We like the second room which has view so we asked the charges and for a second I thought what is it, what have I listened does he (owner Nitish Bhaiya) said only ₹3000. I thought this will be around ₹5k-₹7k because of the view and facilities it has fridge, hair dryer, gas-stove, geyser, and it was a big enough room. After bargaining we settled down to ₹2,700.

Hotel: Beachwood, Near Mall Road, Mussoorie

We were so excited about the room and view. It was too exotic. We take some rest, clicked many pictures, made a funny video. Then we moved for Dalai Hills and Buddha Temple. Although road was in maintainance and slant, but we ride through it easily. We reached and park both bikes for ₹100 at paid parking facility.

You need to climb more than 200 steps to reach the Buddha Statue. Although we reached but I was not feeling well because of my ear pain. I didn't enjoy that moment, didn't took any picture of mine. After stepping down we ate Corn, it was delicious. Stand by there for heat. Then we moved and reached hotel. Now we are tired, and I am in severe ear pain. I talked with Dr. Arvind Uncle as insisted by Papa. He told to take pain killer medicine (Zero Dol SP).

At 08:30 PM, we are ready to go to Mall Road of Mussoorie by walk. We asked the path from Nitish Bhaiya (hotel owner). Didi and Ankit bought blanket for themselves for ₹200 each and I bought my painkiller. After wandering about 30-45 minutes we rollback to the room and slept.

At night around 12:30 AM, the electricity gone down and now we worried about our Bikes because it was parked below the hotel and we couldn't saw it because of no light. Me and Ankit both are worried and continuously peeking through the window to see the bikes but couldn't. Although my pain was vanished by then because of the painkiller. Around 02:30 AM lights came back, I quickly jump from the bed get near to the window to peek the bikes and they are there with no hindrance. We all got relief and slept.

Day 04: Last Ride to Rishikesh

Before check out, we had decided to visit Landour and came till 10 o'clock and check out. The roads to Landour is too slant, it just there is almost no traffic that's why I am able to ride it easily, otherwise it would be harder. We returned before 10 and check out. But before riding to Rishikesh we ordered food at hotel because it was too good to be true. After having the breakfast we moved. It is even harder to go downhill than riding uphill because you have to constantly keep your fingers on clutch and break.

After passing the Dehradun city you will get a very nice highway that's where I touched 113km. It was terrific and awesome. After this speed I got overconfident and started overtaking couple of cars and taxis. Once I increased to 50+ and try to overtake and just then a curve came up where I flicker a bit and rubbed into the middle wall of the road but thankfully I didn't crash. No one's got hurt not even me. Just my shoes got damaged which belongs to Ankit. We stopped nearby and checked if there's any major damage in the bike but there wasn't.

We reached Rishikesh and we need to find a hotel. There is too much traffic and heat. While searching for hotel I lost Ankit way and stopped and wait till they found a hotel. When got it at ₹2,000. I started moving with Map in my phone and phone in my shirt's upper pocket. It took me almost 30-40 minutes to reach hotel. There was too much heat that I have to take bath, can you believe it? We take rest of 1 hour before exploring Rishikesh.

I have only one shirt left to wear which is only for warm climate and I got it in Rishikesh. We visited Triveni ghat by Auto because of the traffic and the decision was actually right. Attend the Ganga Aarti and sang Hanuman Chalisa there. Then moved towards Ram Jhula but we didn't walk on it because we are out of time and the bridge was shivering too much while walking. Then we walked to hotel and slept.

Day 05: Trip Hangover

We packed and check out before 9 o'clock, we reached bike rental shop just before 11 and returned the bike. Bhaiya asked have you crashed anywhere but we rejected.

Now we have 5 hours before our bus to ISBT Kashmere Gate, Delhi. We found a cafe where we ate Aaloo Pratha and sat till 03 PM. Then we took our trolley from bike rental shop and moved to Bus Stand. Got into bus and slept. Then we dinner at the bus stoppage and reached Delhi on time. We took Cab and gone to NDLS railway station. We waited for 5 hours there in waiting room.

Sat on train at 06 AM morning and reached Bhopal on time at 02:20 PM. We took Cab to Dosa shop because hostel has no food at that time. We ate Dosa and slept.

Miscellaneous

There are stuffs I wanted to share regarding this trip.

An Expense Tracker

We are three but only one pays in whole trip and at the end every spends splits and the amount was returned. In Jaipur trip, I paid all the expense and in this Tanu Didi paid. At first I was not going that's I thought about creating an Excel sheet to track our expense. The next day of return I made an Excel sheet, and entered all the expense which was noted down in Notes app by Ankit.

Dehradun Trip Expenses - Excel Sheet

A Chatting Group

It will be very helpful if you create a chatting group for the trip. You can share important common information there. We have created a group on WhatsApp.

Backpack

In my opinion, you should only keep the essential things in your backpack also keep it manageable so that you can carry it easily for long time without fatigue.

Streaming Mode

Anshul Raj Verma — Tue, 10 Mar 2026 10:40:00 GMT

I have been thinking to stream my study sessions on YouTube and want to post videos of my thoughts and thinking too.

I like the way @emmytcnr share her thoughts in a very concise and simple way, nothing fancy or flashy just thoughts.

This way I can also share my thoughts, keep track of my growth, practice English and more.

There is one thing I've to keep in mind that this should be solely to improve myself not to indulge into it. Keep the videos raw and straight forward so that I don't spend more time on editing (maybe you can outsource the editing part).

Testing OBS Studio

I have installed OBS Studio software to stream and record with my MacBook Air M1 laptop. There are couple of settings I have to focus to record videos and following is the best setting I've got:

Input Video Resolution: 3310x2210 (16:10)
Output Video Resolution: 3310x1890 (16:9)
Video Encoder: Apple H.264 Hardware
Bitrate: ~8k-12k for Streaming and ~15k-25k for Recording
Audio: 320 Bitrate and 48kHz Frequency

YouTube recommend the final video resolution should be of 16:9 but my laptop is of 16:10, that's why I have modified the Output Video Resolution.

Something to Learn with Code

I can also learn some topics to automate the Thumbnail creation not with AI instead with presets or templates. I can also explore the field of Instrumental music generation from AI and code.

Live Streaming

I'll use live streaming to learn something with focus and determination as I'll be live it will force me to be on track.

One thing, I need to keep in mind while live streaming is that to not share any private information. I might need to restrict the streaming to a certain application windows like Coding Editor, Terminal, Browser like that.

March Journal

Anshul Raj Verma — Sun, 01 Mar 2026 00:00:00 GMT

Week 10 Journal

Added sitemap in the website.
I have to get crazy for Rust, so that I can learn it.
Learned Rust concepts i.e. Traits, Trait Object, Iterator, Type Coercion.
Created Family Tree data structure with pydantic.BaseModel in Python.
Created Justfile recipes to create new content template for the website instead of JavaScript script.

Week 11 Journal

Tested and started live streaming on YouTube.
- Wrote a blog named Streaming Mode.
- There are some problems I've got like:
  1. I'm not able to watch the live streaming from my Phone. Why? Not even from the mobile browser.
In Rust:
- How is this Ok()/Some() is working instead they should be like Result::Ok()/Option::Some().
- We can’t return a Union type in Rust like in python (i.e. ValueError | KeyError) instead you’ve to use Dynamic Dispatch like Result<(), Box<dyn std::error:Error>>.
Feeling sick but able pull up mindset to not rest and just sit and learn.
Gone to trip to Dehradun with @Ankit and @Tanu from 13 March to 19 March.

Week 12 Journal

Gone to trip to Dehradun with @Ankit and @Tanu from 13 March to 19 March.
Wrote a blog on Dehradun Trip.
Created Expense Tracker Excel Sheet for the recent trip.
Watched movie Dhurandhar 2 in cinema hall on 21 March, in 08 AM slot and returned to hostel around 01 PM.

Week 13 Journal

Watched movie “96”.
@Pratham called to tell me to get ready.
@Anurag’s birthday party on 26 March.
Explored Burn framework again.
Read a nice small article from Kagi Smallweb. If it doesn’t add, it subtracts. This article clearly show my philosophy of attaining/owning lesser materials/things in my life. I am totally happy with my 2 years old clothes or 4 years old shoes. If, it is satisfying my need then it isn’t worth replacing or just bringing new one for the sake of having a newer one.

Sit Down and Ask

Anshul Raj Verma — Thu, 19 Feb 2026 00:40:00 GMT

Come sit down beside me and stay a while. Ask a question to yourself, Why were you walking, climbing, or running?

Now think about one of your closest and ask the same to him/her. If both have similar not if same answer then just take a moment and don't do anything just sit and sit, nothing else. Maybe see your surroundings, people, dogs, sky, or even a blank wall. Just sit and think nothing, you may find relief.

I've done that this noon but the same as it was written instead I was listening to Phoebe Bridgers on headphone and walking to BHEL Ground. I got there and clicked some pictures of trees, flowers, cattles and surrounding.

It was just a escape from loneliness and lone learner. I am skeptical about writing this as I don't think this is a right thing to do I am writing my feel and sharing freely where anyone can read it. I feel frightened.

Hahaha... Where I was started and see where I am now. This happens all the time. I deviate.

I deviate from my routine, projects, studying, calls, almost everything.

SSH with Bitwarden

Anshul Raj Verma — Sun, 15 Feb 2026 00:00:00 GMT

I have switched to SSH from HTTPS for GitHub about two years ago. And that introduced me to the term Secure Shell (SSH). Although, even now I don't fully understand it but I've got a little gist of it.

SSH encrypts your data before transferring it to the server with the generated keys and when the server transfers you the data only the private key is able to decrypt the data.

Manage SSH with Bitwarden

Generate new SSH key:

ssh-keygen -t ed25519 -C "email@example.com" -f "~/.ssh/new-ssh"

The command generates two files
- ~/.ssh/new-ssh: Contains the PRIVATE KEY. DO NOT SHARE THIS.
- ~/.ssh/new-ssh.pub: Contains you public key.
In Bitwarden desktop app, create new SSH Key > Save > Edit > Import the content of ~/.ssh/new-ssh in Private Key field. The Public Key and Fingerprint field will gets update automatically.
Again in Bitwarden, go to Settings > Check "Enable SSH agent".
Add the Bitwarden socket file path in your config (.bashrc or .zshrc) file. Then, restart your terminal.
```
# For Linux/MacOS
export SSH_AUTH_SOCK="$HOME/.bitwarden-ssh-agent.sock"
```
Now, delete the ~/.ssh/new-ssh file. After this you only have the ~/.ssh/new-ssh.pub file.
```
rm ~/.ssh/new-ssh
```
Add the Public Key (content of ~/.ssh/new-ssh.pub) in your GitHub account.

Now create ~/.ssh/config file to add the host configuration.

HostName github.com
 User git
 PreferredAuthentications publickey
 IdentityFile ~/.ssh/new-ssh
 IdentitiesOnly yes

Check where your config works?
```
ssh -T git@github.com
```
Clone GitHub repo like:
```
git clone git@github.com:user/repo
```

How Does Above Config Better?

It is better because the Private Key (~/.ssh/new-ssh file) is not on your system, it is safely and securely encrypted in Bitwarden and your tools are able to access it from Bitwarden SSH Agent. You may check it whether the Bitwarden SSH Agent is running or not?

ssh-add -L

Also your SSH is always available in Bitwarden, you don't need to create new you reset your system. Just copy it from Bitwarden.

`Host` And `HostName` In `~/.ssh/config`

From gemini-3-pro

Think of your ~/.ssh/config file like the Contacts app on your phone.

Host: The Nickname you give the contact (e.g., "Mom"). This is what you type.
HostName: The actual Phone Number (e.g., 555-0199). This is where the call actually goes.

Modify the above ~/.ssh/config:

Host gh
  HostName github.com
  User git
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/new-ssh
  IdentitiesOnly yes

Now you verify your SSH connection like:

ssh -T gh

Or, clone GitHub repo:

git clone gh:user/repo

Here the gh works as the alias for git@github.com

February Journal

Anshul Raj Verma — Sun, 01 Feb 2026 00:00:00 GMT

Week 06 Journal

Talked with @Manik about office relocation in Pune.
Came to Hostel in Bhopal on 2026-02-01.
Discussing the Transfer Certificate and busy in college for this.
I will stick with Astro implementation of the website. Its fun and learn to build it.
- Integrated some of cool and important things in my website i.e. RSS, Open Graph Images, Favicon, SEO, and Sitemap.
- I had a very confusing experience with trailing slashes in Astro application.
End semester exam of @Ankit is ongoing.

Week 07 Journal

Had a conversation on call with @Adityaojas for TC, office relocation, and future.
Almost migrated the main website to Astro.
- Renamed the v2 repo to website-astro.
Migrating the diary repository.
- Will convert docs/page/pages to blog as they are not so distinct.
- All routes has their own OG Image.
- Resume is serving at /resume route.
Reiterated over Account Statement EDA notebook.
IDEA: Create a notebook where you can see your browser history analysed either by LLMs or EDA.
Using rumdl for Markdown linting and formatting in website codebase.
Tried to deploy the website-fumadocs repository on GitHub Pages but couldn’t as I didn’t found any reference. Also, I don’t want to dig into this, now. :(
Moved the website-astro repository to main arv-anshul.github.io repository.

Week 08 Journal

Synced SSH keys with Bitwanden and configured developer accounts with BitWarden’s SSH Agent. Also wrote a blog ”SSH with Bitwarden”.
Added search feature in website with Pagefind. Built it with Antigravity editor.
Taking Audio Course by HuggingFace.

Week 09 Journal

Worked on Syllabus Helper project.
Added notes on some unmaintained repositories:
- arv-anshul/diary
- arv-anshul/campusx-learning
New Biome and Ultracite update messed my linting and formatting of my website repository.
Using song in my Alarm to break my sleep in the morning as I do not pause the song in the middle.
Watched India vs West Indies Super 8 match of T20 World Cup. India won.

The feeling at Platform No. 4

Anshul Raj Verma — Sat, 31 Jan 2026 00:00:00 GMT

This sensation is very horrifying. As I sit at Platform No. 4 of Patna Junction. I remember when tell myself that I love being alone but this moment is feels different.

Currently I am mostly thinking about the dropout process of College.

They are constantly telling me that is too early to decide, you may regret about this decision in future.

But as I had analyzed the pros and cons of the College I came to the decision of dropout.

In past one and half month I have explored so much Burn, Rust, Astro, Fumadocs and more.

I am also thinking about the train reservation and TT as I don't have one, just a general ticket. I need a good rest and sleep.

Docs Conventions

Anshul Raj Verma — Mon, 26 Jan 2026 00:00:00 GMT

I have zeal to follow best possible convention which is fool proof and best, yes best. I know this is a psychological problem than a good habit.

Due to this I am not able to ship fast, I got strangled with best practice and conventions like proper formatting, lifting, error handling, and more.

Although this helps me to learn a better specs but this blocks me from doing the things faster. I chase the best option out there which is not good I guess.

For this I generally try to observe and tackle most of the by using opinionated tools like formatter and linter or by defining conventions for myself.

So What Are Docs Conventions?

Schema for every JSON and YAML files present in data directory.
Frontmatter schema for each and every doc file.
Manage UI components.

Important pages in my docs:

/: Home page
/projects: Page to display all projects.
/about: To display all about me and my experiences.
/friends:
/movies:
/music:
/resume:
/blog:
/diary:
/diary/page:
/diary/journal:

Dotfiles Management

Anshul Raj Verma — Fri, 23 Jan 2026 00:00:00 GMT

This will be the full refactor of my arv-anshul/dotfiles repository from file structure to speak effective configs.

In repository, main config directory will ./home instead of the root of the repository.
Use XDG Base Directories to comply with opinionated convention.
Move the main .zshrc to $XDG_CONFIG_HOME/zsh directory instead of $HOME directory. Also keep one at $HOME/.zshrc but for private or system level use case.
Use AI to get suggestions around best practices and tools.
Use comments to explain configs to recall later.

Currently, using tools such as:

[chezmoi]: Dotfiles manager.
[brew]: Package manager for macOS.

Some other which I will consider in future:

[mise]: Lets you manage envs, tools, scripts, and more for a project.
Podman: Alternative to Docker.

Migrate Docs to Fumadocs

Anshul Raj Verma — Tue, 20 Jan 2026 00:00:00 GMT

I've been planning to migrate my docs from Mkdocs (with Material Theme) to other JavaScript based framework. At first I discovered Mintlify but later found that it is not an OSS, so I had to dump it. Later from openalternative.co, I found out about Fumadocs and its fabulous, perfect. This is what I need.

So I did some digging and it is complex to understand for some new eyes. I've tried all templates Next.js, React Router, and Tanstack Start. I found the Tanstack Start template easy by File structure and Syntax so I stick with it.

My Requirements for the Docs

Supports MDX.
A top navigation bar. It will be better if it also shows icon.
Sidebar navigation with icon (if specified).
Autogenerated Social cards (only if specified, default otherwise).
Supports UI components paradigm.

Previous Migration - `diary/v1.0.0`

Previously, I've migrated my diary from Obsidian to MkDocs and it went well. I've stick with it until, its time to try something new, something better.

Fumadocs Learning

I have chosen the Tanstack Start template for my case due to its simplicity.

File Structure

./
├── content/  # Contains all the .mdx files
├── data/  # Put extra data in structured format to load into app easily
│   └── info.yaml
├── public/  # Contains important public assets
│   └── favicon.ico
├── src/  # Main source directory
│   ├── components/  # Contains UI components
│   │   └── not-found.tsx  # For 404 pages
│   ├── lib/
│   │   ├── layout.shared.tsx
│   │   └── source.ts
│   ├── routes/  # Contains route specific files
│   │   ├── api/
│   │   │   └── search.ts
│   │   ├── docs/  # Related to /docs route
│   │   │   └── $.tsx  # Define layout for "/docs/$" pages
│   │   ├── __root.tsx
│   │   └── index.tsx  # Define layout for "/" route
│   ├── types/  # Define custom types for TypeScript
│   │   └── yaml.d.ts  # Just to load YAML files from /data directory
│   ├── styles/  # CSS files
│   │   └── app.css
│   ├── routeTree.gen.ts  # Autogenerated by Fumadocs
│   └── router.tsx
├── README.md
├── biome.json  # Linter and Formatter
├── bun.lock
├── cli.json  # Specify the Base-UI components
├── package.json
├── source.config.ts  # Define other docs collections i.e. /blog, /projects
├── tsconfig.json
└── vite.config.ts

Define New Collections

You can define new collections other than /docs like /blog or /project. You may read the Fumadocs blog for better understanding.

Define blogPosts collections in source.config.ts file:

import { defineCollections, frontmatterSchema } from "fumadocs-mdx/config";
import { z } from "zod";

export const blogPosts = defineCollections({
  type: "doc",
  dir: "content/blog",
  schema: frontmatterSchema.extend({
    author: z.enum(["Anshul Raj Verma"] as const),
    date: z.date(),
    description: z.string(),
    icon: z.string(),
  }),
});

Parse the output collection in lib/source.ts:

import { blogPosts } from "fumadocs-mdx:collections/server";
import { loader } from "fumadocs-core/source";
import { lucideIconsPlugin } from "fumadocs-core/source/lucide-icons";
import { toFumadocsSource } from "fumadocs-mdx/runtime/server";

export const blogPostSource = loader({
  source: toFumadocsSource(blogPosts, []),
  baseUrl: "/blog",
  plugins: [lucideIconsPlugin()],
});

Define the /blog page layout at src/routes/blog/index.tsx.

import { createFileRoute, Link } from "@tanstack/react-router";

import { blogPostSource } from "@/lib/source";

export const Route = createFileRoute("/blog/")({
  component: RouteComponent,
});

function RouteComponent() {
  const posts = blogPostSource.getPages();

  return (
    <main className="flex-1 w-full max-w-350 mx-auto px-4 py-8">
      <h1 className="text-4xl font-bold mb-8">Latest Blog Posts</h1>
      <div className="grid gap-6 md:grid-cols-2 lg:grid-cols-3">
        {posts.map((post) => (
          <Link
            key={post.url}
            to={post.url}
            className="block bg-fd-secondary rounded-lg shadow-md overflow-hidden p-6"
          >
            <h2 className="text-xl font-semibold mb-2">{post.data.title}</h2>
            <p className="mb-4">{post.data.description}</p>
          </Link>
        ))}
      </div>
    </main>
  );
}

Now, create src/routes/blog/$.tsx to render the blog contents.

import browserCollections from "fumadocs-mdx:collections/browser";
import { createFileRoute, Link, notFound } from "@tanstack/react-router";
import { createServerFn } from "@tanstack/react-start";
import { useFumadocsLoader } from "fumadocs-core/source/client";
import { InlineTOC } from "fumadocs-ui/components/inline-toc";
import defaultMdxComponents from "fumadocs-ui/mdx";
import { blogPostSource } from "@/lib/source";

export const Route = createFileRoute("/blog/$")({
  component: BlogPage,
  loader: async ({ params }) => {
    const slugs = params._splat?.split("/") ?? [];
    const data = await serverLoader({ data: slugs });
    await clientLoader.preload(data.path);
    return data;
  },
});

const serverLoader = createServerFn({
  method: "GET",
})
  .inputValidator((slugs: string[]) => slugs)
  .handler(async ({ data: slugs }) => {
    const page = blogPostSource.getPage(slugs);
    if (!page) throw notFound();
    return {
      path: page.path,
    };
  });

const clientLoader = browserCollections.blogPosts.createClientLoader({
  component({ toc, frontmatter, default: MDX }) {
    return (
      <>
        <div className="w-full max-w-350 mx-auto px-4 py-12 rounded-xl border md:px-8">
          <h1 className="mb-2 text-3xl font-bold">{frontmatter.title}</h1>
          <p className="mb-4 text-fd-muted-foreground">{frontmatter.description}</p>
          <Link to="/blog">Back</Link>
        </div>
        <article className="w-full max-w-350 mx-auto flex flex-col px-4 py-8">
          <div className="prose min-w-0">
            <InlineTOC items={toc} />
            <MDX components={defaultMdxComponents} />
          </div>
          <div className="flex flex-col gap-4 text-sm">
            <div>
              <p className="mb-1 text-fd-muted-foreground">Written by</p>
              <p className="font-medium">{frontmatter.author}</p>
            </div>
            <div>
              <p className="mb-1 text-sm text-fd-muted-foreground">At</p>
              <p className="font-medium">{new Date(frontmatter.date).toDateString()}</p>
            </div>
          </div>
        </article>
      </>
    );
  },
});

function BlogPage() {
  const data = useFumadocsLoader(Route.useLoaderData());
  return <div>{clientLoader.useContent(data.path)}</div>;
}

export function generateStaticParams(): { slug: string }[] {
  return blogPostSource.getPages().map((page) => ({
    slug: page.slugs[0],
  }));
}

Write your blogs at content/blog/my-blog.mdx.

Integrate Social Cards

I will use Takumi for Social Cards generation (because its written in Rust). You may refer to Fumadocs Takumi integration docs.

How to configure the Takumi OG image generation in my Tanstack Start + vite (SSR) app? I am using bun.

Icons in Docs

By default, Fumadocs gives support to Lucide Icons which is nice but I also want Simple Icons for brand icons.

For this I have to create custom plugin.

Follow a Routine

Anshul Raj Verma — Fri, 16 Jan 2026 00:00:00 GMT

Everyone has 24 hours in their days, but when you follow a routine you optimize thyself not only for that day in fact for the others too.

As I am watching @emmytcnr YT channel I feel I should create a routine. Not so strict, instead it will be like instructions to for me to what to do in the time you've got in that day.

It will be available in Google Notes with checkboxes.

The routine majorly consists of the things which are important and repetitive I want to do in a day. This will help to keep track of them:

Reminders like wake up time, exercises, time of sleep, book reading, coding timing, etc.
Things you may want to prioritize.

This will help me to evaluate myself on daily basis, also makes me more disciplined.

GitHub Account Flagged

Anshul Raj Verma — Wed, 14 Jan 2026 00:00:00 GMT

On 08 January exact 08 AM, I got a mail to reset my GitHub account due to suspicious activity. I am like what, how, who? But I certainly did as being said. And I thought that's it and it's all good now.

At same evening I raised an issue in other's public repo. I thought it was a success but when I get back into that repo's issue tab I am not able to see my issue. So I thought it might be got missed (which didn't happened before), I may raise another but I thought I should check once to confirm.

Somehow, I figured out that it was opened but none can see it except me. Then my mind tricks, it may have happened by that today's suspension.

I go to mails' Trash tabs (because I deleted that email), restores it and read it again from start word-by-word. I checked account's security logs there nothing suspicious expect that suspend, unsuspend, and oauth key terminations.

I reached to GitHub Support and raised a ticket to resolve this issue.

They said to create new account and use it. I was like what, why? What did I done wrong? Tell me.

They said the GitHub System suspected that my account is used distribute malware. :salute:

I insisted and told to review my account manually, there is nothing suspicious. This account is important for me. I haven't created another because I know I haven't done anything wrong.

They said it was your arv-anshul/dotfiles repository which is convicted and you should either delete it or privatize it. So I changed the visibility of the repository to PRIVATE.

And hence, the support team cleared the restrictions from my account. :finally:

Thanks.

See email thread

Review of 2025

Anshul Raj Verma — Tue, 13 Jan 2026 00:00:00 GMT

I have already did the review of my 2025 Journal.

But that's the review of my Journal there are many things I haven't mentioned or even thought to write about. The things like How I got into the College, and most important (I even once thought about writing this) The Roller Coaster Life in Bhopal.

Below are some points which I am able to remember while writing about Year 2025:

Ignoring LeapX internship in favor of 12th board exams, also not preparing for the exams.
12th board exams completed in February and barely passed.
After the news, got panicked and started searching for college, even though before the result I don't want to go.
Eventually, I got admission via @Tanu with hell lot of drama, thrill, and suspense.
@Ankit also took admission there and now it's time to move to Bhopal.
Me and @Ankit both decided to move there one month prior of starting of new classes. Took train reservation of 3rd July.
That first month in Bhopal is fun and exciting, there is no pressure we are just waiting for college.
Then the college started and we made some very, very, very, good friends @Anurag, @Himanshu, @Diwas, @Ankush, and @Krish. We just met about a few weeks ago and its feel like that we are friends from childhood :).
As I was started to learn what and how does the college teach, its getting clearer that, this is not what I am here for. They are just revising the school routine (even stricter). "आसमान से गिरा, खजूर में अटका"
There are many incident and story from the college and would not be good for this, so skipping.
As the time passes by, it's getting more prominent I am not going to survive this college environment. Whatever I've learned so far is not going evolve. Because I am not practicing, not learning, not even able to extract time for it and this frustrate me.
I discussed this situation with my parents and they said if you don't like it, don't do it, but make sure you are not making any mistakes.
At the end on 23rd December, I came home and confirmed that I am not going to dropping out and will learn and study from other sources.
Currently, I am in my house, on my bed, under the blanket, time: 09:55 AM.

What's Left

I will be writing these article soon to complete the review of 2025 and link them here:

The Roller Coaster Life in Bhopal 2025
The Roller Coaster Life in College 2025

Podcast #3 by Arunesh Don

Anshul Raj Verma — Mon, 05 Jan 2026 00:00:00 GMT

YouTube randomly recommended me this video (Podcast #3 by Arunesh Don) on my PC and it instantly caught my attention (because of its channel name) but at that time I saved it into Watch Later playlist. Now on 05 January at 02:30 AM I watched it. While watching I got some thoughts and also thought to email them to the Creator but couldn't so I am writing it at evening.

Thoughts

Beginning: At start, I was thinking that this is just another video of an Indian student doing random shit with their friends. And it was, if you watch the video half way.
Observations: I observed the narrator’s command in his surrounding, its not as usual as I thought it would be. In fact he has very grip on them the boys don’t argue, disgust, unnecessary laughing or making fun out of words which can happen many times in my case/scenario.
Surroundings: I assume that the room in the video is of narrator’s and I have to say it is full of a mess but it has very motivating, inspiring, rebellious AURA too. Because the hostel room belongs to a B. Tech. student but it has geographical maps, motivational posters, and a poster written “SLAVE?” on it.
Interruption: There was an interruption in the conversation which were having by someone who needs to do some photo-copy print. At first, he denied and told them to come after 30 minutes but eventually did the work. And in that process there was a guy who entered the room and get to his bed (I assumed) to do something but the narrator instructed that guy “Don’t you read the instructions on the door?”. Then that guy instantly get out of the room and knock the door and enters (that’s what I observe).
Ending: In the end, there were almost two to four sentences which hits me and I also agree with them.
1. While criticizing be a dominant, rigid, strict, confident person also maintain eye contact but while appreciation be a wise, gentle, clam person.
2. Agar jo tu aaj hai, aur wahi cheez tu kal bhi follow karega to log tujhe predict kar lenge.
3. Don't be only a lovable person, be a dominant one too.
4. more...

More insights on Arunesh

After the video finished, I started looking for his email id in channel's about section couldn't found it. Instead I came to know about that his channel has almost 280+ videos which are from multiple topics like motivation, fraud complains, criticism, etc. (which is very bold, IMO). His videos focuses on self improvement and society improvement. He is completely on different level, he is blunt and straight forward :saluting_face:.

Although, I've got his email from one of his video (mobile no. too) but I want to prepare to send an email.

Will watch his other videos specially his podcast series.

January Journal

Anshul Raj Verma — Thu, 01 Jan 2026 00:00:00 GMT

Week 01 Journal

Came home on 23 December.
Learning and exploring CNNs concepts like Object recognition, object detection. Also recalling fundamentals like filters, channels, padding, pooling, etc.
IDEA: A CV project for student attendance using Object detection technique from video input.
Explored Burn and actually understood something (but not much).
- Wrote couple of lines and able to build a pipeline to train a simple NN model with Iris dataset.
Performed cleanup in:
- GitHub Stars: Removed dead, un-useful (personally) repositories.
- Folo RSS: Replaced some rsshub:// feeds with open-source maintained projects like RSSBridge, GitHubTrendingRSS. Also added rss feeds.
Hero Hockey India League 2026 is streaming for free on YouTube 😉.

Week 02 Journal

Disabled Prettier for dprint in Zed editor.
- The dprint extension in Zed is not maintained wisely as the language configuration is not full featured. The maintainer copied the configs of biome-zed extension’s config.
I stumble upon a very nice video on Object Detection by YOLOv8 by “Murtaza Workshop” YouTube channel. Now I can easily work with YOLO models and can build some CV projects.
Thinking of creating content or documenting my learning journey or my projects on YouTube. That’s why I installed OBS Studio to record and stream videos.
- Requested for live streaming on my YouTbe account. Let’s see how it goes?
- First I will start non-face cam live sessions, only audio.
I sometime write the Journal in Raycast Notes so the sentences might become present tense as I am writing and thinking/doing at same time or going to do.
There’s a /blog in main website which I use for technical notes and references. Now, I want to write my thoughts, random small thoughts (if I get time for it). So I’ll use new /diary/page endpoint for that.
- I am tweaking the /blog for better visibility or for no reason.
Updated and fixed Zed editor VS Code Dark Modern theme. Put it in my dotfiles.
I got suspended by GitHub in the morning and just after automatically resumed, but why? I don’t know. Although I got an email about it but there is no clarification. They just instructed to reset the password.
- Now, my hosted websites (arv-anshul.github.io) are not working.
- I am able to commit into my repositories and raised an issue in other repository successfully.
- My issues are not visible in public, checked from incognito.
- I have created a new ticket (#3999489) at GitHub Support on 2026-01-08T23:29:00.
Successfully, configured mdformat in arv-anshul/diary repository to format the Markdown files as expected.
- Not able to do so for arv-anshul/arv-anshul.github.io repository due the formatter's behavior of replacing   (like HTML tokens) with corresponding UTF-8 character, which I don't want. There is no option to disable it.
SG Pipers won Hero Hockey India League (Women) tournament and I'm supporting them ;).

Week 03 Journal

Amazing video on Model Distillation for Query Intent Classification. He doesn’t distill any LLM or SLM instead a simple-small NN using the data generated by GPT 3.5 model.
Wrote a new page Review of 2025.
Explored codebase of Presenton, to learn how to create an AI to generate or edit PPTX documents. And I have to say there is a lot going on not only in Python (Backend) side in fact on TypeScript (Frontend) side too. I want to review it once more soon.
IDEA: A CLI which fetches release notes of provided GitHub repositories and displays the notes in terminal.
- Now I am learning Rust programming through this.
- Check rlog in arv-anshul/thrust repository.
Oooh! Got the full access of my GitHub account. Support assisted to privatize to delete the arv-anshul/dotfiles repository, I changed the visibility to PRIVATE. See diary page to know in detail.
Ordered Google Arcade gifts. I got a T-Shirt, a Water Bottle, some Magnetic stickers, and an Arcade Stickers Sheet.
- Although, I may have been gotten some more but I got distracted by Jaipur trip and then College.
Wrote a new page Follow a Routine.
I am planning to migrate my website (from mkdocs-material) to some other framework maybe Mintlify, Astro, etc. First I'll do arv-anshul/diary then eventually (if I like it) migrate arv-anshul/arv-anshul.github.io.

Week 04 Journal

Continued learning Rust from rlog project.
Working on the work given by @Sarvesh.
Explored Fumadocs for personal docs migration alternative.
- It is quite intriguing but hard to figure out what’s happening in the files. All the functions are scattered, but now able to navigate. I know they are in there but with new eyes it's hard to figure out.
- Read page Migrate Docs to Fumadocs.
Installed Zorin OS on home laptop (DELL) it is working fine for now, but the laptop’s display is too bad to stare, not good as Mac’s.
Store SSH Keys in BitWarden.
Used LocalSend for file transferring between Phone and Mac.
IDEA: Create a web app (specially static web app if possible) to convert a photo into passport sized photo and the convert it into A4 size PDF for printing.
Updated dotfiles. See page Dotfiles Management.
Came Patna for JEE test of @Akansha.
Stayed in Patna for three days (24 Jan to 26 Jan).

Week 05 Journal

Kalinga Lancers won HIL tournament. I supported them ;)
My priorities:
- Website
- Learn the Burn framework
- Excel Deep Learning
- Improve the yt-watch-history
- Or new project.
Worked on website migration (Fumadocs + Tanstack Start + Vite + Bun). Currently I am not very confident to migrate.
Trying to book train ticket to return with @Himanshu.
Trying Astro to create the website from scratch. I got some progress with it and I am liking it, planning to stick with it.

Journey of 2025

Anshul Raj Verma — Wed, 31 Dec 2025 00:00:00 GMT

At the end of the year, I wrote this so that one can get a summary what have I mainly done throughout the year on monthly basis. I haven't mentioned all the moments from the Journal only whom I found important.

January

Keywords: BERT, Keras 3, HF Transformer, Fine-tuning, LaTeX, Journal Summarizer, LaTex, Resume, Hockey

Summary

Explored finetuning the BERT models with PyTorch and HuggingFace Transformer libraries with efficient methods like PEFT, LORA, etc.
Created the Journal Summarizer in Marimo for this diary repo.
Worked with LaTeX and related tools to create a pipeline for resume management.
Got interested in Hockey (from Hockey India League) and created a web scraping project which scrapes HIL matches data from the official website.

February

Keywords: Kaggle Courses, Reels, AI Agent, Job Applier Agent, Browser Use, Kunkun

Summary

After reading the journal I just came to know that I scrolled REELS for procrastination :WoOoW:
Meet with @Sagar for a Job Applier AI Agent with Browser Use.
Enrolled in Agents Course by Hugging Face and Intro to LanGraph by LangChain.
Contributed in Kunkun project but now there is almost no development ongoing.

March

Keywords: Coding Hero, LangGraph Studio, RAG, unsloth-ai, Opik

Summary

Learning LLM Agents creation with LangGraph also learning related concept like RAG, few-shot-prompting, etc.
Came to know about unsloth-ai for easy and efficient LLM finetuning.
Contacted with LeapX.ai and restart the work with them but the split happened.

April

Keywords: Rust for ML, RSS, Folo, Post Generator, Cousin Wedding

Summary

Explored Rust ecosystem for Data Science and ML.
Found Folo app for RSS Feed reader.
Worked on new idea by @Aditya Post Generator for Linkedin but dumped.
Took break for cousin wedding.

May

Keywords: spotyhive, ty, College

Summary

Created a program to fetch and store my Spotify Data like Playlist and Album data.
ty: new Python type checker by Astral.
Got busy in finding College after passing 12th.

June

Keywords: College

Summary

Got admission in LNCT Bhopal in B. Tech. (CSE/DS) course.
Did nothing as I was busy in College and stuff.

July

Keywords: College, Typst, @Araadhay, Infina Research, Password Manager, BitWarden

Summary

Moved to Bhopal for College.
Helped @Araadhay in his Infina Research project.
Started using BitWarden for password management.

August

Keywords: Google Arcade Program, Typst, Jaipur Trip, College, College Orientation Program

Summary

Joined into Google Arcade Program.
Migrated to Typst for Resume management.
Jaipur Trip with @Ankit and @Tanu.
College is started by 25th August.

September

Keywords: Google One, MCP, Google Jules, Gravatar, j178/prek, Zyou SDK

Summary

Claimed Google One plan for students for 12 months.
Deleted useless repositories from GitHub account.

October

Keywords: EDA, Diwali, Chhath Puja, Burn

Summary

Performed EDA on Online Transaction data.
After giving the MidSem-1 Exam came Home in Chhath Puja.
Wanted to learn Burn framework.

November

Keywords: Zensical, College

Summary

Zensical is now replacement of Material for MkDocs.
Trapped in College works so there is no entry.

December

Keywords: Helium, College, Exams, Liberation Notes

Summary

Installed Helium browser.
Came Home after MidSem-2.
Being liberated from College.

Liberation Notes

Anshul Raj Verma — Wed, 17 Dec 2025 00:00:00 GMT

I have been stuck in deciding whether should I pursue my Graduation or drop from it for Self-Study. It is like mood-swings, one moment I think about dropping out and another I thought, should I really because I can do many thing with my friends, even though college education doesn't gives me much?

Pursue Graduation	Self-Study
❌	✅
Make more good friends	You are all responsible for any consequences
Make organic connections with interactions	You will be on yourself, no one is there to tell you what is good right.
Do something crazy with friends (first they need to teach themself)	Be discipline and learn alone
Student benefits (Github, Google, etc.)	You can create something good everyday/everyweek
The degree and placement cushion after 4 years	You can learn and explore new things everyday/everyweek

Things I Wanted to Learn and Explore

Data Science and GenAI

Deep Learning
Transformers and LLMs
- Working
- Fine Tunning (LoRA, QLoRA, etc.)
MLOps
Advance GenAI frameworks and concepts (LangGraph, MCP, AI Agents)
Tools like Prefect, DataLakes (DuckLake, DataBricks, DeltaLake, LakeTower),

more...

RUST programming language
- Axum: API development in Rust (Try both, learn one)
- Actix: API development in Rust
- Burn: Deep Learning framework in Rust
GO programming language
- API development in Go
Typst
- Thumbnail or Card with a certain ratio creation with Typst. Something like OG Image Playground.

Extra

How to write story?
How to film a scene?
How to take photos?

Future Plans

December Journal

Anshul Raj Verma — Mon, 01 Dec 2025 00:00:00 GMT

Week 49 Journal

Install Helium browser as the data of ARC's data erased while doing some testing.

Week 50 Journal

Late for entry.

Week 51 Journal

Busy in exams.

Week 52 Journal

Came Home after MidSem-2 on 23 December.
It's decided that I can take DROPOUT now, I can be get out of the DILEMMA. So, need to follow and update the Liberation Notes.

November Journal

Anshul Raj Verma — Sat, 01 Nov 2025 00:00:00 GMT

Week 45 Journal

MkDocs is now deprecated and the maintainers are building Zensical. See the blog.
Doing very less and distracted. Busy in College study and work.

Week 46 Journal

Late for entry.

Week 47 Journal

Late for entry.

Week 48 Journal

Late for entry.

October Journal

Anshul Raj Verma — Wed, 01 Oct 2025 00:00:00 GMT

Week 40 Journal

Performed EDA on Online Transactions data.
Add logging and tests in Zyou SDK.

Week 41 Journal

Finally, migrated the resume repo to Typst.

Week 42 Journal

Surprisingly, yt-comment-sentiment project works fine even after ~10 months means the model is available on DasgsHub repo.
Performed EDA on Account statement pdf file.

Week 43 Journal

Diwali vacation.

Week 44 Journal

At home.
Wi-Fi is very disturbed here.
Wanted to study Burn framework but couldn't due to some error in Zed editor. Also there is no tutorial around it on YouTube but there is a Burn Book though.

September Journal

Anshul Raj Verma — Mon, 01 Sep 2025 00:00:00 GMT

Week 36 Journal

Got the Google One Plan for 1 year (till 1st Sept 2026).
Working with MCP with @Aditya.
Hard to cope-up with Google Arcade Program.
Implemented a custom Validator class in Pydantic.

Week 37 Journal

Built MCP server using arv_zyou project at Intership.
Built web-app for spotyhive repo using Google Jules.
Updated Gravatar profile.
Explored a pre-commit alternative written in Rust j178/prek.

Week 38 Journal

Implementing Narwhals into Zyou SDK @Aidtya’s project.

Week 39 Journal

Deleted multiple useless and senseless repositories from my GitHub account.

August Journal

Anshul Raj Verma — Fri, 01 Aug 2025 00:00:00 GMT

Week 32 Journal

Joined Google Arcade Program.
Migrating resume repo to Typst instead of LateX.
5-Day trip to Jaipur. (9 Aug to 13 Aug)

Week 33 Journal

Progressing in Google Arcade Program.

Week 34 Journal

Progressing in Google Arcade Program.
@Manik asked to scrape 99acres.com properties.
Connected with @Aditya to discuss his new project based on Facebook Ads, MCP, Redis.
Attended Orientation Program of College on 23 Aug, 2025.
Tried Activity Watch application to track my screen time on Mac but couldn't used it long because it is too verbose and I don't need that much as it is too much for my use case.

Week 35 Journal

Trying to claim Google One for students.
Working on zyou SDK for @Aditya.
Talk about internship with @Aditya.

Redirect Search Engine

Anshul Raj Verma — Mon, 28 Jul 2025 00:00:00 GMT

Approach

Get user's query.
Search with minimal engines like DuckDuckGo, Google Search, Bing, etc. and get their most relevant result.
Finally redirect to the top resultant URL.
Alternatively, we can also recommend most relevant results in dropdown menu for ease of redirection for user.

Why Someone Use this Search Engine?

Due to ease of website navigation from traditional search engine.

Cons

Someone can use search engine like Perplexity to get its query's result in natural language response.

Questions

Is this a good product to build?
Is this a good product to sustain in future?
Is this product really help peoples in their daily life?
Is this product really outsmart current competitors?
What are more cons of this product from POV of current competitors?

July Journal

Anshul Raj Verma — Tue, 01 Jul 2025 00:00:00 GMT

Week 27 Journal

Done Nothing.

Week 28 Journal

Done Nothing.

Week 29 Journal

Continuing to write Weekly Journal from now on.
After thinking about a day or two, finally decided to work on job-finder project.
Discovered Typst, now using it in job-finder project to compile CV in PDF and PNG format.

Week 30 Journal

Got an idea to use HTML & CSS for templating in carousel-ai project as in HTML we just need to fill the placeholders and export it in PDF format. First tried to create the template in Typst but couldn't; LOL.
Helping @Araadhay in his project related to legal paper querying and summarization. He has created a demo web-app using Lovable and also wrote a backend script.

Week 31 Journal

IDEA: A search engine which can redirect to the website according to the user's query. (rejected after talking with LLM)
Created new secondary email address aanshulrv@gmail.com.
Exploring password manager world to ease my social account management.
Now using BitWarden as my new password manager and authenticator app (replacing Google Authenticator app) in MacBook and Android device.
SnapDrop is deprecated; now PairDrop is new project.
Helping @Araadhay in his InfinaResearch project.

June Journal

Anshul Raj Verma — Sun, 01 Jun 2025 00:00:00 GMT

Week 23 Journal

Done Nothing.

Week 24 Journal

Done Nothing.

Week 25 Journal

Done Nothing.

Week 26 Journal

Done Nothing.

May Journal

Anshul Raj Verma — Thu, 01 May 2025 00:00:00 GMT

Week 19 Journal

New repo spotyhive to store Spotify Data in JSON format.
Checked about ty and pyryefly Rust based Python type-checker.

Week 20 Journal

Marimo let you import python function from notebook.
Archived repository ColdTurkeyBlocker-Pro against ColdTurkeyBlockerPro-Activator-FREE.
Got the 12th board result, now busy in exploring college.

Week 21 Journal

Done nothing in code, busy in prepration for College and stuffs.

Week 22 Journal

Done Nothing.

April Journal

Anshul Raj Verma — Tue, 01 Apr 2025 00:00:00 GMT

Week 14 Journal

Read about data science and machine learning libraries in Rust ecosystem. Libraries like polars (for Data Manipulation), linfa (for Machine Learning) and ndarray (like numpy) are there.
I am planing to create an ML model in Rust and convert it into WASM to run it on browser, although I am not sure that I am actually going to achieve this but I will definitely read about it.
Deep dive into pydantic learned advance validation and serialization with custom functions while working on job-finder project.
I have not been able to work this week with LeapX as I am not feeling well.
Trying to run FLUX-dev model for Chili style image generation but doesn't because my MAC's RAM got out of memory.
Trying dprint to format docs, previously tried with mdformat-mkdocs but it couldn't gives satisfiable result, although dprint doesn't supports MkDocs but want to give it a try for basic formatting.

Week 15 Journal

Performing advance analysis on Campaigns datasets with @Aditya Sir.
In sage-backend, we now infer LLM model from prompts' metadata.
Gone to Patna for sister's JEE test met with @Shubham Bhaiya.
Office re-location talk with @Aditya Sir.
Got disturbed by heavy storms as the electricity and internet connection is fluctuating.

Week 16 Journal

Re-exploring the RSS Feeds trying to gather different important news at one place with Folo app.
@Aditya Sir thought about new SAAS idea that to generate PDFs for social media posting specially LinkedIn.
Updated introduction paragraph on website and GitHub profile. Used LLMs to generate it from highlight points.

Week 17 Journal

@Aditya Sir started working on Post Generator for LinkedIn (using AI).
Taking break for wedding.

Week 18 Journal

No work took break for cousin wedding from 25-04-2025 to 07-05-2025.

March Journal

Anshul Raj Verma — Sat, 01 Mar 2025 00:00:00 GMT

Week 10 Journal

Explored interrupt and Human in the Loop mechanism in LangGraph.
Joined in some classes of Coding Hero platform its good to grow your network but good for learning as the teachers are learner itself and have no experience of teaching. (Just my POV)
Learning LangGraph and creating a POC with Query IPL DB project idea. Just to get familiar with all concepts.

Week 11 Journal

Exams are over and now continuing to learn and create.
Explored LangGraph Studio web version.
Building ipl_agent but intrigued by the performance of agent as it doesn't perform as excepted and give random answers. I need to learn how to improve the results of LLM like using few-shot-prompting, better RAG, good prompting technique, and more.
IMO, finetuning a LLM with unsloth-ai is easy as there are plenty of resources to learn how to finetune LLM using unsloth-ai. I think you just need to find or create a dataset for your purpose and you are ready to go.

Week 12 Journal

Re-join LeapX.ai as Data Science and AI Intern currently stipend and duration is not decided.
Explored project repos of LeapX.ai and they are messy af, so I've added pre-commit and used uv to manage it.
Forgot to write Week 12 (and almost Week 13) journal on time so I am writing it on last day of Week 13.
Found a nice and beautiful alternative for Swagger UI called Scalar.com, although it is beautiful but not straight-forward as Swagger.

Week 13 Journal

Discussed about job-finder project with @Dipanjan.
Wrote Dockerfile for production as well development stage for LeapX's project.
Explored opik to manage prompts and log LLM traces for LeapX's Sage project.
Started working on job-finder project and already implemented some major features. See the project repo at @arv-anshul/job-finder.

Job Applying Agent

Anshul Raj Verma — Thu, 06 Feb 2025 00:00:00 GMT

Sagar from CampusX group

Requirements

Requirements	Description	Extra
Framework	Langchain	We will eventually try to drop it. But if this is a showcase project then a framework is best.
Job Opening Data	Jobs where agent will apply	Maybe agent will fetch data on-the-go by web scraping.
User Login (LinkedIn)	Need to login the user on the platform in order to apply

Agent Workflow

LinkedIn Profile Scraper: Scrape LinkedIn profile of user and get relevant info.
Parse User's Resume: Parse info from user's Resume.
Retrieve relevant jobs: On the basis of user's parsed info agent will retrive relevant jobs for user after analyzing jobs' description.

LLM Agent Task	Detailed Process	Requirements/Dependencies	Considerations
Receive & Parse Job Data	- Accept job listing details (title, description, requirements) via API/webhook or direct scraping.<br>- Parse and structure the data for further analysis.	- Access to structured job data.<br>- Data parsing modules (e.g., regex, NLP parsers).	- Handle variations in data format.<br>- Ensure data completeness before processing.
Extract Key Job Attributes	- Analyze job description to identify required skills, qualifications, and responsibilities.<br>- Highlight keywords and competencies to match user profile.	- NLP techniques for entity extraction and keyword analysis.<br>- Domain-specific dictionaries or models.	- Handle ambiguous or vague descriptions.<br>- Ensure extraction accuracy.
Align with User Profile & History	- Retrieve relevant parts of the user’s resume and past experiences.<br>- Map job requirements with user skills to determine fit and emphasize strengths.	- Access to the user’s structured profile data.<br>- Matching algorithms to align job needs with profile attributes.	- Maintain user data privacy.<br>- Address gaps between job requirements and user profile gracefully.
Generate Custom Application Materials	- Construct a tailored prompt for generating a cover letter (or resume adjustments) using the extracted job and user information.<br>- Generate draft content via the LLM.	- Access to a capable LLM (e.g., GPT-4) with prompt engineering.<br>- Template management and customization frameworks.	- Balance personalization with professionalism.<br>- Verify that generated content is factually consistent with the user’s data.
Iterative Refinement & Quality Check	- Run generated content through self-evaluation steps (e.g., re-prompting for clarity, checking for tone, grammar, and job-specific relevance).	- Additional LLM prompts for self-review.<br>- External grammar and style-check tools if necessary.	- Allow for multiple iterations to refine quality.<br>- Optionally incorporate human feedback or pre-defined quality metrics.
Simulate Application Interface Integration	- Prepare the final content to be inserted into LinkedIn’s application forms (e.g., cover letter text, resume attachments).<br>- Format the content to meet form requirements.	- Knowledge of LinkedIn’s input fields and formatting constraints.<br>- Integration layer with the application automation module.	- Ensure compatibility with the LinkedIn interface.<br>- Handle edge cases like character limits or specific formatting needs.
Execute Automated Submission	- Pass the finalized content to the automation module (e.g., Selenium script) for form submission.<br>- Monitor confirmation messages or error responses.	- Stable API/web automation scripts.<br>- Communication channel between LLM agent and submission module.	- Robust error handling is critical.<br>- Confirm that submission was successful and log the outcome.

Generated with ChatGPT.

Problems

Applying to Jobs in Behalf of User.

If we are going to implement this then this might be the heaviest situation because we have to deal with multiple steps like:

User Login.
Migrating to each job listing page.
Applying to job by filling the form including uploading user's Resume.

Due to this problem we should first focus on creating an AI agent which find relevant jobs for user by seeing their LinkedIn profile and their Resume.

Agent Flow

LinkedIn profile URL.
Resume as PDF
A tool will extract info like experience, job type, job mode, and more.
A tool will scrape relevant jobs on the basis of user's extracted data.
Sort the scraped jobs by most relevant for user.

LangChain Perception

How to use messages? (With .stream method)
How to implement async tools? (For profile scraping and resume parsing purpose)

🔎 Job Finder - Raycast Note

Current Plan

We planned to provide a platform where both Employee and Employer list their skills and requirements and then we will provide an interface where both Employee and Employer will finder their match on the basis of their skills and requirements.

Problem (with Current Plan)

As we are no one who is going to use product the idea Employer listing is hard to be gain success on, so I decided to drop it. (read next)
We can scrape websites like LinkedIn, Naukri, etc. and gather data related to Jobs and make the platform only for Employee (as name suggests Job Finder).

Platform Usage

User will upload its Resume/CV on our website.
We will parse their CV using OCR and extract a structure information for our use case.
We will then asks the user to fill a partially filled form to get full info about the user.
The form is already filled by the info we extracted from user’s CV.
Then, we will recommend most similar Jobs we got in our database.

Future Plans

We may also provide a service to create a resume by filling a simple form.

References

jsonresume.org: For custom resume creation.

Resourses

February Journal

Anshul Raj Verma — Sat, 01 Feb 2025 00:00:00 GMT

Week 06 Journal

:star: Dipanjan cited me in his MBA's final year project documentation.
Learning from Kaggle courses. And I have to say they are promising from begginers perspective.
- Completed Intro to Deep Learning course.
Updated website's and diary's docs with new updated syntax of Admonition and more.
@Sagar (from CampusX group) suggested to build a LLM agent which can apply in Jobs on platforms like LinkedIn. So, creating its HLD.
Nowdays, I am Full on PROCASTINATION MODE because of 12th board exams and also not able figure out what to do next. Just killing time on reels, web series and youtube.

Week 07 Journal

Meeting with @Sagar and we discussed about:
- Job applier on LinkedIn but he wants to apply from official website instead.
- Suggested to look into browser-use tool which is apparently interesting and useful.
Explored browser-use it is interesting but there are some points to note:
- Works on heavy prompting.
- Very nice approach/feature of Controller.
- As usual problem of LLM apps very dynamic in nature of completing tasks.
Enrolled into Hugging Face Agents Course.

Week 08 Journal

Exams are here.
Completed Unit 1 of Fundamentals of Agents course by HuggingFace.
Completed Intro to LangGraph course by LangChain.

Week 09 Journal

Explored Kunkun app.
Built an extension for Kunkun, Search Emoji.
Cloned Kunkun and did some UI tweaks. Created PR#219 there.
[ WIP ]
Added GPG Key in my GitHub account to sign the git commits and tags.
Found new font SF Mono which is by Apple Inc. itself but didn't know about it yet but I've installed and it and using it from Homebrew font-sf-mono-for-powerline. Its just it lacks visually appealing italic variant.

Fine-Tune Transformers

Anshul Raj Verma — Sun, 05 Jan 2025 00:00:00 GMT

Learning how to fine-tune a BERT model using PyTorch/TensorFlow from HuggingFace for your use case is an art in itself because there are so many ways and methods to do. And you will not able to figure out easily which is the best for your use case. By the way, you can always refer to HuggingFace documentation.

For Example!

Choose between PyTorch and TensorFlow. (lets choose PyTorch)
If you are importing your dataset with pandas or polars then need to create a custom class by inheriting torch.utils.data.Dataset class.
Then need to tokenize the data and need to use DataLoader and Data Collator.
Then use a for-loop to train and validate the model.

But there is an easy way of fine-tune, by using objects like transformers.TrainingArguments and transformers.Trainer which reduces the manual looping complexity.

Fine-Tune Process

Load Data

Import dataset your method such as pandas, polars or other ways.

Preprocess Data

Process the data and check the labels. Docs Preprocess Data.

Train-Val-Test Dataset

Split the data into train, validation, and test data. Before doing this you have consider many things like:

How to tokenize the data with certain padding, truncation, max_length, return_tensors, etc.?
Do you need to shuffle the data? (Only shuffle train dataset)
Which object you will use to store the data? (datasets.Dataset or torch.utils.data.DataLoader)
Representation or data type of labels column? This will be different for different type of problems you have to make sure that the data is in correct format.
Is DataCollator required?

Tokenize Data

You need to tokenize the data before sending it to model to trained on. It is done using respective model's tokenizer. You can tokenize the data separately or in batch (recommended).

Padding and Truncation

Batch Creation

You have to cast the dataset into an object which supports the batching.

Data Collator

Data collators are objects that will form a batch by using a list of dataset elements as input.

Data Collator

transformers.DataCollatorWithPadding

Load Tokenizer

A tokenizer is in charge of preparing the inputs for a model. The library contains tokenizers for all the models. Most of the tokenizers are available in two flavors: a full python implementation and a “Fast” implementation based on the Rust library Tokenizers.

Load Model

A pre-trained model which we are going to finetune using our custom dataset.

BERT Model

Model Training/Finetuning

PEFT Methods

PEFT offers parameter-efficient methods for finetuning large pretrained models by training a smaller number of parameters using a reparametrization method like LoRA and more.

Evaluate Model

Model Prediction/Inference

Training with PyTorch

You can either fine-tune a pretrained model in native PyTorch or with transformers.Trainer class (recommended).

Read this documentation by HuggingFace "Fine-tune a pre-trained model" where they explain how to fine-tune a pretrained model using both the methods separately.

Also refer to this tutorial by same team for "Text Classification".

January Journal

Anshul Raj Verma — Wed, 01 Jan 2025 00:00:00 GMT

Week 01 Journal

Learning how to fine-tune BERT model using PyTorch from HuggingFace example.
- Sentiment Classification: transformers_sentiment_wandb.ipynb
- Multiclass Classification: transformers_multiclass_classification.ipynb
Planning to leverage the power of Keras 3 which is written in their docs.
- Its hard (or not possible) because HF's transformers library doesn't support it but I have checked that keras_hub library has it's own BERT model there which I can use, I think.
Will explore more in next week:
- [x] How to fine-tune BERT using PyTorch + HF Transformers?
- [ ] How to fine-tune BERT using Tensorflow + HF Transformers?
- [x] ~~How to fine-tune BERT using Keras3 + PyTorch + HF Transformers only?~~ (NOT POSSIBLE)
- [x] What are these terms like SPDA, PEFT, LORA and more for fine-tuning purpose.

Week 02 Journal

Explored new method to finetune BERT model using transformers.TrainingArguments and transformers.Trainer classes.
Wrote a ref page where important articles are listed for finetuning BERT model.
Commiting all the finetuning codes in the form of python script and marimo notebook in @arv-anshul/notebooks repo.
Explored PEFT methods like LoRA. Tried to finetune BERT model using them.
Found a way to format my mkdocs docs. See @astral-sh/ruff repo to know more about formatting mkdocs docs.
- .pre-commit-config.yaml
- .markdownlint.yaml
Planning to migrate all /ref pages to /blog because these aren't any different from them. Also, I don't write blogs because I feel the articles are better fit for /ref pages which reduced the usages of /blog pages.
- Migration maybe broke some links that I will fix later.

Week 03 Journal

Finally, committed the Journal Summarizer as marimo app for this diary repo.
:star: Migrated /ref pages to /blog in website.
Also tried to format markdown files in website's repo using mdformat-mkdocs but it's not acceptable for me.
Raised issue in mdformat-mkdocs repo.
Worked on creating my resume (CV) using LaTeX (.tex format). And tried to manage it with python script.

Week 04 Journal

More work on resume (CV) management. Explored LateX and pdflatex like tools for resume management.
New repo @arv-anshul/hockey. Scrapes data related to Hockey from altiusrt.com websites using scrapy framework. And publish the data on Kaggle.
Wrote a Kaggle notebook on sentiment classification on Reddit comments dataset using Keras.
Exploring deepseek-r1 distill models with smolagents for nlp-to-sql tasks.
Hectic exploration of LaTeX tool with Podman. (literally, very hectic)

Week 05 Journal

Created @Suraj's college PPT on Introduction to Systems of Human Body at his place.
New repo @arv-anshul/resume.
@Nitish sir (@CampusX) announced Gen AI Course for free on YouTube.
Joined Kaggle Competition S05E01.
Feeling clueless with next step around anything (project, learning).

Dockerize - FastAPI - UV

Anshul Raj Verma — Fri, 27 Dec 2024 00:00:00 GMT

Let's dockerize my yt-comment-sentiment project which is a FastAPI app managed via UV.

Official Docs

FastAPI docs for Dockerization
UV docs for Dockerization
UV-FastAPI example docs for Dockerization

:thinking: Then, Why this?

I also consider above docs to fulfill my requirements but this includes:

Multistage build to reduce final image size.
Best and Flexible practices to use uv in Docker.
Solution of some problems which I have encountered.

`Dockerfile`

I am taking reference of my yt-comment-sentiment which I have developed and recently and continuously improving it.

FROM python:3.11-slim AS builder
# See official docs at https://docs.astral.sh/uv/guides/integration/docker/#available-images
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/

# Install gcc for wordcloud
# Using `wordcloud` in backend and it require `gcc` package to build wheels to work in python.
RUN apt-get update && apt-get install -y gcc && apt-get clean

WORKDIR /app

ADD pyproject.toml uv.lock /app
# Install dependencies with `--extra=backend` dependencies
RUN uv sync --extra=backend --frozen --compile-bytecode --no-install-project

# Copy only necessary files/folders to reduce image size
COPY params.yaml /app
COPY backend /app/backend
COPY ml /app/ml

RUN uv sync --extra=backend --locked

# Final stage
# Use multistage build to reduce the image size.
FROM python:3.11-slim AS final
COPY --from=builder /app /app

WORKDIR /app

# Run backend using fastapi-cli
CMD [".venv/bin/fastapi", "run", "--host", "0.0.0.0", "--port", "8000", "backend/app.py"]

If you get any other problem please refer to official docs.

YouTube Comment Sentiment

Anshul Raj Verma — Fri, 27 Dec 2024 00:00:00 GMT

You can read about this project's each segments in details in their respective pages at ML Side, Backend Side and Frontend Side.

After reading above pages you'll get to know about:

What had I used?
How did I done?
I have also shared some common mistakes you could encounter while building this.

Frontend Screenshots

YT Comment Sentiment - Backend Side

Anshul Raj Verma — Fri, 27 Dec 2024 00:00:00 GMT

Technology	Description
	A Python library for building and training machine learning models.
	A collaboration platform for machine learning, hosting data and MLflow models.
	A platform to manage the ML lifecycle, including model tracking and deployment.
	A modern web framework for building APIs with Python, known for its speed.
	A Python library for data validation. Used to validate API data.
	A testing framework for Python, used to test the FastAPI application.
	An API to access and manage YouTube video data, including comments.
	A cloud platform for hosting APIs, websites, and applications.

What I Followed to Know?

[!IMPORTANT]

As I am learning Python, Data Science and Machine Learning for more than 3 years. I don't have to look around to learn new things to build this. This part is kind of easy for me.

But as I said earlier, the documentations and ChatGPT is most important resources you can onto. :wink:

Need to get an API key from Google Developer Console to interact with YouTube Data API.
Need to create an account on DagsHub to store/track MLFlow experiments and models.
Created a DVC pipeline to run the MLFlow experiments seemlessly using dvc repro command.
After creating the FastAPI app, I've used pytest to test it and also setup a pre-commit for it.
Deployment on render.com.

What Type of Problems I Have Faced?

Render.com

As I have used uv to manage my project but render.com doesn't support uv out-of-the-box so I have used pip to use uv for dependencies installation.
```
pip install uv && uv sync --extra=backend --compile-bytecode
```
Also, render.com only serve apps on port under $PORT env (which 10000 most of the times) so make sure to explicitly provide while running app through uvicorn or fastapi-cli CLI.
```
# For uvicorn
uvicorn run --host 0.0.0.0 --port $PORT backend.app:app

# For fastapi-cli
fastapi run --host 0.0.0.0 --port $PORT backend/app.py
```

Docker

I am using wordcloud to create a plot in a FastAPI route. While building docker image FROM python:3.11-slim image, I am getting error because wordcloud package needs gcc package to build wheels. So you need to explicitly install gcc before install wordcloud as python package.
```
# Install gcc for wordcloud
RUN apt-get update && apt-get install -y gcc && apt-get clean

# Now install project dependencies including wordcloud
# ...
```
Also use multistage builds in Dockerfile to reduce the image size. See uv docs.

YT Comment Sentiment - Frontend Side

Anshul Raj Verma — Fri, 27 Dec 2024 00:00:00 GMT

Technology	Description
	A fast and efficient package manager for JavaScript projects, known for its disk space usage.
	A customizable component library for building elegant UIs in modern web applications.
	A utility-first CSS framework for creating custom designs quickly and efficiently.
	A next-generation frontend build tool for blazing-fast development and hot module replacement.
	A progressive JavaScript framework for building user interfaces and single-page applications.

What I followed?

[!IMPORTANT]

Documentation is the most important resource for your learning and I followed it thoroughly.

Took help of ChatGPT to solve bugs and asked questions related to these frameworks/tools.

Followed only one YouTube video to learn VueJs.
Used shadcn/vue as components library. Just followed the docs.
Whole build_frontend.yaml Github Action workflow written by ChatGPT, isn't it amazing :exploding_head:.

Although, I didn't have much knowledge of Vue, Vite and ShadCN like frameworks/tools but their documentation and ChatGPT helped me so much that I am able to learn, build, diagnose and deploy the frontend in almost 2-3 days.

Yes, some credits goes my past knowledge of programming because that's why I able to figure out how do things works and how to handle them by doing right things.

What Type of Problems I Have Faced?

हे भगवान! While learning and working on this frontend project, sometime I get messed up with very silly typo mistakes in JavaScript.

The things you are building happily in local will totally change when you are trying to deploy it and I have faced this too :cry:.

VueJs

How to emit data from Child component to Parent component in VueJs?
- GFG Article
- Vue School Article
- Also take a look into official docs to know best practices for this.
How to create and work with forms?
How to use v-model?

Vite

While deciding how to setup proxy in Docker because I've setup it using Vite and it works like charm in local but while writing Dockerfile I am able to figure out the solution.
- Proxy is used in VueJs YouTube video and thats why I have followed it.
- See this docs
While handle environment variables in nginx.conf file in Docker environment. I have used ChatGPT read articles but didn't get to solution.
How to work with different .env files in Vite.

December Journal

Anshul Raj Verma — Sun, 01 Dec 2024 00:00:00 GMT

Week 49 Journal

:star: Internship ended at LeapX after 5 Months.
- They asked to continue after exams.
- I decided to go there to meet them.
Continue learning RNN (LSTM, GRU, Deep RNNs and BiDirectional RNNs) from CampusX.
Trying to study for 12th boards but its hard to focus on it because (I think) its a whole new genre to study from scratch (because I forgot all the things).
Created a Jupyter Notebook around Search Keyword Classification which I had work on while in internship.
- Tackle all the problems like Category Classification, Intent Classification, Score Assigning and Similar Keywords Grouping.
:cat: Discovered this interesting project on top of Stylus browser extension which apply Catppuccin Theme on various popular websites such as GitHub, OpenAI, Perplexity, Spotify, and many more. Isn't this interesting to transform all the website theme in one color palette.
- After this I am considring to try this Catppuccin Theme but I am afraid that I would like it because I don't really like it's color pallete.

Week 50 Journal

Exploring LLM agents with Langchain & Ollama. Also trying to built something around specially which I have already built in internship.
Worked on notebooks/llm/chat-with-data scripts.
Explored python's tree-sitter schema for Zed in PR#21389.
Is development of Zed editor very slow. @Hitesh Choudhary stated this in his recent video?

IMO yes, because if you look closely it seems they are more focused on AI side of it. One of the most important feature like builtin ipython notebook is not their priority RN which is bad for most of the users (including me), I constantly switching with VSCode for that.
Marimo is working on html-wasm export feature which is actually nice.
- Marimo has multiple hidden capabilities you just have explore it (you'll love it, i am sure about it).
- Just waiting for the support of VSCode's builting notebook layout.

Week 51 Journal

:technologist: Worked on yt-comment-sentiment project.
- Explored pre-commit and pytest while working on it.
:pirate_flag: Found an amazing website fmhy.net where linksof many piracy sites were listed.
:man_facepalming: I accidentally deleted a file (where test for a FastAPI app is written) using rm command and then I had wrote it again from scratch.
Update social links on website and more. See arv-anshul.github.io repo commits.

Week 52 Journal

:star_struck: Building frontend side of yt-comment-sentiment project using VueJs.
Learning VueJs for frontend, as I don't know TypeScript using only Javascript.
Deployed both Backend on Render.com and Frontend on GitHub Pages.
Also wrote about yt-comment-sentiment project.
I have decided to make this project big enough to be called as a product in itself. (दृढनिश्चय)
I also think to write a marimo notebook which summarize my development/learning/inconsistency using LLMs. As I want to summarize my whole year diary/journal with this idea.
Wrote a blog on how to dockerize a FastAPI app which is managed by uv, took reference of yt-comment-sentiment project.
I have also learned many things around pre-commit and pytest which I have used in project recently.
Installed ghostty terminal emulator and it is great, looking forward to stick with it.

YT Comment Sentiment - ML Side

Anshul Raj Verma — Mon, 04 Nov 2024 00:00:00 GMT

Data Handling Steps

Data Gathering

For comment's sentiment prediction we need a data which has Comments and its corresponding Sentiment. And for that we have used dataset used in the course.

Data Preprocessing

Preprocess by lowercasing the words.
Cleaned the texts by removing stopwords and punctuations.
Applied lemmetization using WordNetLemmatizer.
Then, stemming using PorterStemmer.

EDA

Checked target column's distribution.
Performed intensive EDA by creating many additional features using comment's chars, words and sentences.
Generated wordcloud to see different sentiment's frequent words.

Model Building Steps

Comment Vectorization
- Before transforming performed some basic preprocessing steps on comments like lowercasing, lemmetization and stemming to make vectors more consistent.
- Evaluated multiple vectorization methods like BOW and TF-IDF.
- Also, performed hyperparameter tuning on vectorization methods by tuning parameters like n_gram and max_features.
- Chosen TF-IDF Vectorizer model to transform comment texts into vectors which passes into ML Model.
Feature Engineering
- Created multiple new features using comments' texts like word count, etc. which help the model to learn the comments' sentiment better.
Hyperparameter Tuning
- Used Bayesian Optimization Technique to perform hyperparameter tuning on models.
- Tuned models on their most important parameters.
- Logged best parameter of each models with MLFlow to evaluate further.
Evaluation
- Used MLFlow UI to check which model is performing well on the dataset.
- Evaluated on:
  1. Overall accuracy_score
  2. Different sentiment's r1_score, precision and f1_score value.

November Journal

Anshul Raj Verma — Fri, 01 Nov 2024 00:00:00 GMT

Week 45 Journal

No work at LeapX because busy in Chhat Puja.

Week 46 Journal

@Aditya sir asked to access company's email and GitHub while I am not with my laptop. So, they changed my email's password and get the work done.
Update data fetching script in campaign-analysis-api project with @Shubham.
- They are migrating DB from MongoDB to Amazon RedShift.

Week 47 Journal

Updated cron job to store campaigns data into MongoDB (hourly).
- Used loguru package for logging purpose and its awesome out of the box.
Added copies and region wise breakdown dataframe in ad-campaign-bot/caht-with-data project.
Started working on yt-comment-sentiment project (from DSMP course).

Week 48 Journal

Worked on youtube-comment-sentiment project where I have integrated MLFlow and DVC and it is actually working as expected.
Started learning and practincing Deep Learning.
- Learning from CampusX and Alexander Amini (introtodeeplearning.com).
- CampusX has started new playlist around PyTorch.
- Also, noticed Keras 3.0 for its backend compatiblity.
Understanding RNN and trying to implement it from scratch with youtube-comment-sentiment project's dataset.

October Journal

Anshul Raj Verma — Tue, 01 Oct 2024 00:00:00 GMT

Week 40 Journal

Migrate my dotfiles to chezmoi.
:star: 2 months of extension in internship at LeapX.ai.
Tried axum in Rust.
Learning Rust from @piyushgargdev YouTube course, he teching from "The Rust Book".

Week 41 Journal

Came to Jabalpur on 9th October.
Actively working with LeapX, remotely.

Week 42 Journal

Stayed in Jabalpur and worked with LeapX.
Had a 1hr long chat with @Dipanjan.
Moving to PUNE from Jabalpur on 19th October.

Week 43 Journal

Working with LeapX (@Haresh, @Saif and @Aditya) on Advance Analytics Dashboard APIs.
Trying to meet some people in Pune but I didn't met due to work and lack of ...

Week 44 Journal

Storing campaigns data into MongoDB from FB APIs. Also solving some ongoing problem while storing hourly data.
Recommended @Dipanjan to @Aditya.

Canvas AI

Anshul Raj Verma — Sun, 15 Sep 2024 00:00:00 GMT

Inspired by '1GbJQ7fHgqo'

Prompt Examples

These are some samples which I have used to verify my AI and be sure that it is working fine. You can see the images and their respective structured responses (responses are written inside code-blocks, just below each image).

Example 1

{
  "expression": "x=3;y=4;x+y=?",
  "result": "7",
  "explanation": "The sum of x and y is 7"
}

Example 2

{
  "expression": "(x + y)^2",
  "result": "225",
  "explanation": "Substitute the values of x and y in the expression and simplify using BODMAS."
}

Example 3

{
  "expression": "10 km/hr * (50 meters / 1000 meters/km) * (3600 seconds / 1 hour)",
  "result": "18 seconds",
  "explanation": "The car is travelling at 10 km/hr and it has to cover 50 meters, so we can calculate the time it takes to cover the distance."
}

{
  "expression": "10 km/hr * 50 meters",
  "result": "13.89 seconds",
  "explanation": "First convert 10 km/hr to meters/second, then divide 50 meters by the speed to get time."
}

Example 4

{
  "expression": "√(10² + 10²)",
  "result": "14.14 m",
  "explanation": "The diagonal of a square forms a right-angled triangle with two sides of the square. Using Pythagoras theorem, we can calculate the diagonal."
}

Example 5

{
  "expression": "pi*3^2*12, pi*3^2*5",
  "result": "360pi m^3, 45pi m^3",
  "explanation": "The tank is a cylinder with radius 3m and height 12m. The volume of a cylinder is pi*r^2*h. The filled water is also cylindrical with radius 3m and height 5m."
}

September Journal

Anshul Raj Verma — Sun, 01 Sep 2024 00:00:00 GMT

Week 36 Journal

Now, Polars supports altair charts with .plot accessor.
Heavy refactoring in google-ads-api project.
Finally, migrated housing-dashboard project to Taipy from Streamlit.
Helped @Sambhav with his Job Assignment related to simple RAG project.
Learned how to use F-Droid and also installed some utility apps.

Week 37 Journal

Getting some issue with DELL keyboard but now it is fixed.
:technologist: Will not continue the internship after 3 months instead work on a big project to incorporate all my learning there and make it a standout project.
- Read "After LeapX" thoughts.
Added one more dataset in ad-campaign-bot (Chat with campaign data) project, i.e. age_gender_df. Used Facebook Marketing API docs to understand the process and resultant data from the API.
The ad-campaign-bot or campaign-bot-api project can drop the facebook-sdk package from its dependency and only use httpx to make request with API because we are not taking the advantages of that SDK anyway instead we are just making raw request to the API.
I have got the Mass Driver Font from assets.runno.dev domain.
- Regular: https://assets.runno.dev/md-io/md-io-regular.woff2
- Italic: https://assets.runno.dev/md-io/md-io-italic.woff2
- Bold: https://assets.runno.dev/md-io/md-io-bold.woff2
- I haven't got the "Bold Italic" variant. :disappointed:
Created chrome extension to modify css of pre-defined websites. I was thinking about this for 1 years.
Downloaded the crack version of Aldente (Charger Limiter App) from macbed.com and in order to make it work I have to block Aldente's outgoing connections, so for this I have installed Lulu app.

Week 38 Journal

After watching a YouTube Video, I've created a new project called canvas-ai which can analyse image containing mathematical expression or problem and return a structured response. My first personal GenAI project and its awesome :star_struck:.
New project from LeapX to scrape many-many different webpages to build RAG models. Already scraped Facebook Marketing API docs and submitted to @Shubham.
Used clap crate in md_badges cli tool.
- You can also download the md_badges cli tool in your system as binary with cargo install command, see project's README.md form more info.
:sneezing_face: Ill for last two days :sneezing_face:

Week 39 Journal

Replace Pylance with basedpyright in VSCode.
Successfully configures Zed IDE with basedpyright and ruff extensions.

I think now I can try to ditch VSCode editor.
Enabled Codeium AI completion in VSCode.
Trying Zed and Marimo inplace of classic apps.
:star: New project started in DSMP 2.0. It is a Chrome Extension for YouTube to perform sentiment analysis on video's comments.
Connected with @Mohit and @Dipanjan to collaborate on new DSMP 2.0 project and had a nice discussion around it.

August Journal

Anshul Raj Verma — Thu, 01 Aug 2024 00:00:00 GMT

Week 32 Journal

Finetune the DistilBert Model from HuggingFace for search-keywords classification.
:handshake: Me and @Pushpam met with @Vishal (LeapX) and had an amazing talk at lunch.
Created a collab notebook which further finetune pre-trained LeapX's Keywords Classifier Model with more data and automatically pushes it in HuggingFace models repository.
Learned more about LangChain's LCEL and re-built the campaign-bot from scratch using it.

Week 33 Journal

Scheduled a cron for properties-scraper project on GitHub Actions.
Made changes in campaign-bot streamlit app and also deployed it on HF Spaces.
Switched from bash script to python script to run the properties-scraper.
Created md_badges project in Rust.
Update mkdocs-material for website but this introduces some minor bugs due to removal of Microsoft icons from SimpleIcons repository.

Week 34 Journal

PiyushGarg started new playlist around The Rust Book (online version).
uv v0.3.0 allow support for uv init, uv add, uv remove, uv run and many more. Read the release blog.
Aditya sir posted on housing-dashboard project on LinkedIn, he named it LeapX Localyzer.
New google-ads-api project to setup new GoogleAds campaign where keywords are choosen using ML algo.
Updated GitHub profile README inspired from @twpayne.

Week 35 Journal

Migrating housing-dashboard from Streamlit to Taipy.
Completed 2 Months at LeapX.ai.
Bought new smartphone "Nothing Phone 2a".
Writing my journal in draft form in Google Notes.
Deployed google-ads-api project at Render.com, after a long & hectic bugging session with @Pratham.
:package: uv new release 0.5.0 brings support for build-less packages. So now I can completely ditch rye.
Migrated to uv in diary and arv-anshul.github.io repos from rye.
Talk with @TanuKumari regarding college and SIH project.

LeapX - Internship

Anshul Raj Verma — Sat, 13 Jul 2024 09:13:00 GMT

Joined LeapX.ai as Data Science Intern from 27 June, 2024 for 3 months.

LeapX Projects

Project done while working with LeapX as Data Science Intern.

Properties Scraper

Project scrapes data from Housing.com website and dump into MongoDB after basic transformation using Polars, I've used Scrapy to scrape data. Wrote a python script which automate the process of scraping multiple city in one go.

Housing Dashboard

A dashboard which shows affluent areas of a city after applying clustering on the real estate data of the city.

Uses Taipy framework to create dashboard.
Fetch properties data from MongoDB.
Perform clustering and plot circles on map of selected city with its affluence rank and other metrics.
Deployed on Hugging Face Spaces.

:llama: Ad Campaign Bot

A Facebook Ads Campaign Bot powered by OpenAI LLMs where users can query their campaign related questions.

Bot is flexible enough to answer user's query on multiple camping data format such as daily or hourly campaign data.
Bot can also display graphs/plots related to each query (with some limitations).

Campaign Dashboard

Streamlit dashboard to show insights around Facebook Ads Campaign using Data Analysis and charts using campaign metrics like CTL, CPM, CPC, CTR and more.

Refactored code given by @Aditya.
Deployed on Hugging Face Spaces.

After LeapX

After working for 3 months with LeapX, there are some points I've want to discuss:

Just discontinue the internship and start focusing on personal projects and further learning and try to gather in-depth knowledge around Deep Learning concepts and more.
Continue the internship and start learning deep learning from CampusX side-by-side which will be a win-win situation best utilization of the time.

July Journal

Anshul Raj Verma — Mon, 01 Jul 2024 00:00:00 GMT

Week 27 Journal

Created new GitHub account with arv@leapx.ai email to work with LeapX. (But I don't get this.)
- There is no sense of creating new account just to commit in company's github organization.
Learning many things while working with LeapX.ai.
- Finished the first interface of two paged streamlit dashboard.
- Started working on new project which uses openai and llama-index and it is a kind of AI Dashboard.
- Aditya said that "Team members are liking my work".
- Wrote scrapy scrapper from scratch to scrape Housing.com and pushed to LeapX.ai GH-ORG.
- Wrote multiple parser to parse html pages (using selectolax) of government website's pages where RERA registered agents were listed.
Watched movie "The Whale".

Week 28 Journal

Finally, found the Astral's uv official docs from their GitHub repo's deployment section. The docs are present at astral-sh.github.io/uv.
Watched Love Life (2022) movie.
Binam bought Macbook Air M2 (8GB - 256GB).
Fixed many bugs in arv-anshul/dotfiles repository while setting-up Binam's macbook.

Internship at LeapX.ai

2024-07-08

Started refactoring the big-messy-code (where all dfs were stored in st.session_state) and pushed changes to a new repository campaign-dashboard.
Try scraping data of different major cities like Gurgaon, Delhi, Hyderabad, Mumbai, etc. but failed due to blocking. So Aditya, asked Pritam (USA based guy with better knowledge) so he told "use slow and randomized scraping using Docker".

2024-07-09

🤒 Took a day-off due to fever and loose motion.

2024-07-10

More refactoring in campaign-dashboard (almost completed).
Downloaded MongoDB Compass in laptop to connect with LeapX cluster where scraped data form Housing.com get stored.

2024-07-11

Created HuggingFace account with arv@leapx.ai ID.
Learned how to use HF Spaces and created a GH Action in campaign-dashboard repo to push code from GH to LeapX's HF Space where the Streamlit app gets deployed. (done whole thing in 1 Hour :scream:)

2024-07-12

Learned and added Scrapy Pipeline to move scraped data into MongoDB cluster; also added Data Transformation (using Polars) code for scraped data.

Week 29 Journal

Maintaining the explanation or description of projects done with LeapX as Data Science Intern in thoughts folder.

Internship at LeapX.ai

2024-07-15

Modified housing-dashboard so that it can ingest data from MongoDB.
Scraped 10k+ data of Delhi and Gurgaon city using properties-scraper.
housing-dashboard is running on HuggingFace Spaces.

2024-07-16

Discussed properties-scraper to automate it with @Anurag.
Modified properties-scraper codebase for the same.
[WIP] Also wrote code to scrape Maharastra RERA Agents.

2024-07-17

Meeting with @Pratham for campaign-bot to transform it into an API.
Meeting with @Vishal & @Aditya for housing-dashboard problems.
Discussed workflow and API architecture of campaign-bot API and created flowchart diagram for the same.

Currently assigned Projects to @Anshul

Hi Aditya, I am Anshul, currently assigned with almost 4 projects and it's hard to manage and discuss all of them with
different team members. It will be very helpful if you prioritize these projects so that I can work with them
efficiently without taking too much stress.

**Projects Description**

1. `properties-scraper`: Wrap the project with Docker so that it can scrape multiple cities properties data using `cron`
   job. There are many modifications occur in this project for this like dynamically load `URL` and `city` name. I've
   already discussed about it with @Anurag.

2. `housing-dashboard`: Yesterday (2024-07-17) in a meeting with @Aditya and @Vishal; we discussed some problems
   regarding this project and decided to make some major changes in this project.

3. `campaign-bot`: Yesterday (2024-07-17) meeting with @Pratham; we discussed about creating an API system around this
   to seemlessly integrate this bot in frontend.

4. `campaign-dashboard`: Recently refactored this project (code given by @Aditya) and deployed on HuggingFace Spaces but
   project has very high latency due to bad code management. So, this project also need improvements to handle latency
   and needs better architecture.

Best Regards, Anshul Raj Verma Data Science Intern

2024-07-18

Meeting with @Pratham and @Aditya for campaign-bot API.
There is a little progress in campaign-bot code but getting unexpected result after passing user_prompt in query_pipeline.
Pipeline is not working as expected and there are more concerns like how to retrive data for visualization and pass it as API response.

2024-07-21

Building campaign-bot-api with @Aditya and @Pratham and almost completed it. Made with FastAPI with JWT auth (learned on-the-go).

Week 30 Journal

Droping the idea of journaling of LeapX.ai work.
Learning LangChain and RAG for internship projects.
Discussed about User's Search Intent classification with @Gagandeep & @Vishal.
- What is meant by search intent?
- Identify user search intent with machine learning.
Completed my first month at LeapX.ai as Data Science Intern.
- LinkedIn Post

Week 31 Journal

Team dinner at LeapX.ai. They order meal for me too :partying_face:
Worked on same project Search Intent Classifier project whole week. Trained distilbert-base-uncased model 2 times.
:medal: First time following and loving olympics matches and athlete :saluting_face:
- :star_struck: Loving Lakshya Sen and Hockey Team performaces.
Using and kind of loving the new alternative of Jupyter Notebook; Marimo.

LeapX - Interview

Anshul Raj Verma — Fri, 14 Jun 2024 17:00:00 GMT

Week 24 Journal

Aditya reached me > conducted a meeting on Microsoft Teams > ask questions related to projects, education > other two co-founder joined the meeting > ask to solve a problem in Real Estate domain

Problem Statement

On what parameters we can classify a city with different expensive/affluent region?
How can we cluster/classify expensive region of a city?
How can we bound/decide a region as an affluent region?

Points for Explanation

We can consider city's mean area (sq ft.), mean price, mean price (per sq ft), BHK, properties count.
An area become expensive due to the availability of some commodity in those areas like hospitals, banks, luxury restaurants, gym, airport, hotels, and more. These commodities increases the price of those areas very high.
If the area has some major development project (by government or private) going to take place.
We can consider point number 2 and 3 but if you want to gather those type of data then it is very difficult.
- For example, if you want the data of luxurious restaurants then you have to decide a threshold which defines luxurious restaurants and what about data availability of all restaurants present in city. This thing hard to gather for other commodities as well like airport, pubs, banks, and more.
For extreme cases, we can consider crime rate, transportation, highway, medical facilities, political intervention/advantage, community tension, etc.
We can also check current affairs of those which we are going recommend after the classification of Model because if the current situation in those is not good then we must not consider those areas.
As we can see above parameters are hard to gather for a ML Model but we can build a clustering algorithm which can cluster similar properties and from that

Apply Clustering

If we apply clustering on properties we can say that "This property in this area is similar to that property in that area".

Apply Classification

We have not much parameters through which we can classify areas.
We have to take many majors to evaluate the model because we don't have enough parameters to classify them.

[!NOTE] Aditya's Thought on Above Explanation

This is good thought process, I'd further like you to emphasize on how you would benchmark certain landmarks, high-end cafes, high street markets, expensive residential societies etc. that represent affluent and wealthy localities/radius.

I'd like you to think of a methodology, and basis that, represent those radius on a map.

We can take Gurgaon for an example.

Decide Affluent Region

We should first gather data about those landmarks which are considered as luxurious/expensive in Gurgaon.
Then calculate distance between them and societies (we are considering to cluster/classify as affluent).
Then assign a benchmark/luxury value (between 1 to 10) to those landmarks which create a sort of association between societies and landmarks.
~~Then create a function which aggregate "distance between societies and landmarks" and "benchmark values".~~
Then gather comprehensive data about these societies like property prices, rental prices, people annual income, etc. which tells us about societies current value among other societies. These parameters create distinction among societies itself.
Then apply K-Means Clustering on the whole gathered data and then plot the clustering metrics like silhouette_score to determine the model's wellness.
:ADDITIONAL PARAMETERS: We can extract societies and landmark rankings and ratings from different real-estate websites which further enhance our model's performance.

This process help us to cluster similar luxurious societies and then we can infer or bound affluent area in the given city.

landmark > lat long > radius (2 km) > localities > sort on price (per sq ft) localities (lat long) > landmarks >

Decide on the Basis of Sector

I have all properties data.
Cluster affluent properties.
Group by on localities/sector.
Now you know how many affluent properties are there in each localities.
Why localities/sector; not a certain radius of area because whole city is divided into localities/sector which means

If you want to mark an area as affluent then that are must be in any sector which means that sector is affluent.
If any sector has many expensive properties and or that sector is around luxurious landmarks then we can declare that sector as affluent area.
We can create two clustering model one for properties and another for landmarks (this also help us to identify).
how can we

Decide with DBSCAN

After applying DBSCAN all the propeties were clustered as noise data point.

Ultra luxury (lat long) > find closest property > find both's average >

Intro to Deep Learning

Anshul Raj Verma — Mon, 10 Jun 2024 00:00:00 GMT

Machine Learning VS Deep Learning

As the shape of data increases the ML models cannot able to capture its underlying patterns but deep learning algorithms capture the complex relationship very well.
ML algorithms uses different techniques to learn patterns from data like linear line, spliting criteria, etc. but Perceptron is the building block of DL algorithms which helps to capture almost every patterns of the data.

Perceptron

Perceptron the building block of Deep Learning.

\sum = w_1x_1 + w_2x_2 + ... + w_nx_n + b

Now after calculating the #!math \sum you will use a activation function #!math \varphi which is applied to the weighted sum to introduce non-linearity.

For example, #!math \varphi can be a step function whose output is either 0 or 1.

Important Points on Perceptron

Perceptron split the data in two classes.
Perceptron creates a lines in 2D, plane in 3D and hyperplane in 4D onwards.
Perceptron's geometric intuition is very similar to Linear Regression algorithm.
Perceptron is limited to classify only linearly (or sort of linear) separable classes.

Limitation

Perceptron only works on linear data it doesn't learn non-linear data because perceptron is a linear model which draws a line/plane/hyperplane on datasets.

Multi-Layer Perceptron (MLP)

A Multi-Layer Perceptron (MLP) is a class of feedforward artificial neural networks that consist of multiple layers of nodes in a directed graph. Each node, except for the input nodes, represents a neuron that uses a non-linear activation function. MLPs are capable of learning complex patterns in data, including non-linear relationships, making them widely used in machine learning tasks like classification, regression, and feature extraction.

By using a single perceptron, we are limited to learning only linear decision boundaries. This restricts its ability to model more complex datasets with inherent non-linear relationships. To overcome this limitation, we can add more perceptrons in a structured way to create a Multi-Layer Perceptron.

Why Don’t We Use Only One Layer Instead?

A single-layer perceptron can only model linearly separable data. For example, tasks like distinguishing between two classes in XOR logic cannot be achieved by a single-layer perceptron, as its decision boundary is inherently linear. However, real-world datasets are often non-linear in nature.

Adding hidden layers in an MLP allows the network to transform input features through non-linear activation functions, enabling it to create complex decision boundaries. These hidden layers progressively learn higher-order features, making MLP a universal approximator of functions, as proven by the Universal Approximation Theorem.

How Does MLP Capture Non-Linearity?

MLPs capture non-linearity through two primary mechanisms:

Non-Linear Activation Functions: Non-linear activation functions like ReLU, Sigmoid, or Tanh introduce the ability to model complex patterns in the data. Without these functions, the MLP would behave like a linear model, regardless of the number of layers.
Layered Structure: Each hidden layer processes inputs to extract increasingly abstract features. These features, when passed through activation functions, enable the network to learn representations that are non-linear transformations of the original data.

By stacking layers, the network learns a hierarchy of features, where lower layers might capture simple patterns (like edges in an image), and deeper layers capture more abstract representations (like shapes or objects).

Forward Pass in MLP

The forward pass is the process of passing input data through the network to compute the output predictions. It involves the following steps:

Input Layer: The input data is fed into the network as feature vectors.
Hidden Layers: Each hidden layer applies a weighted sum of inputs followed by a bias term and an activation function. Mathematically, for a hidden layer ( l ):

Hidden Layers in Notation

h^{l} = f(W^{l}h^{(l-1)} + b^{l})

where:

#!math W^{l} is the weight matrix for layer #!math l.
#!math b^{l} is the bias vector for layer #!math l.
#!math f is the activation function (e.g., ReLU).
#!math h^{(l-1)} is the output from the previous layer.

Output Layer: The final layer produces predictions, often applying a specific activation function (e.g., softmax for classification or linear activation for regression).

from keras import Sequential
from keras.layers import Dense

# Define a simple MLP model
model = Sequential([
    Dense(16, activation='relu', input_shape=(4,)),  # Input layer
    Dense(8, activation='relu'),                     # Hidden layer
    Dense(1, activation='sigmoid')                  # Output layer
])

model.summary()

The forward pass results in the computation of predictions based on the current weights and biases.

Backward Propagation in MLP

The backward propagation algorithm is used to train the MLP by adjusting weights and biases to minimize the error between predicted and actual outputs. It works in the following steps:

Compute Loss: A loss function (e.g., Mean Squared Error for regression or Cross-Entropy Loss for classification) measures the error between the predicted output and the true labels.
Backpropagation of Errors: The error is propagated backward through the network using the chain rule of calculus to compute the gradient of the loss function with respect to each weight and bias. The gradients for each parameter are computed layer by layer in reverse order.
Update Weights and Biases: Using the computed gradients, the weights and biases are updated using an optimization algorithm like Gradient Descent or its variants (e.g., Adam, RMSprop):

Backward Propagation with Notation

W^{l} = W^{l} - \eta \frac{\partial \mathcal{L}}{\partial W^{l}}

b^{l} = b^{l} - \eta \frac{\partial \mathcal{L}}{\partial b^{l}}

where:

#!math \eta is the learning rate.
#!math \mathcal{L} is the loss function.

import torch
import torch.nn as nn
import torch.optim as optim

# Define a simple MLP model
class SimpleMLP(nn.Module):
    def __init__(self):
        super(SimpleMLP, self).__init__()
        self.fc1 = nn.Linear(4, 16)  # Input to hidden layer
        self.fc2 = nn.Linear(16, 8) # Hidden to hidden layer
        self.fc3 = nn.Linear(8, 1)  # Hidden to output layer

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.sigmoid(self.fc3(x))
        return x

# Instantiate the model, loss function, and optimizer
model = SimpleMLP()
criterion = nn.BCELoss()  # Binary Cross Entropy Loss
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Dummy data for training
inputs = torch.randn(10, 4)  # Batch of 10 samples, 4 features each
targets = torch.randint(0, 2, (10, 1)).float()  # Binary targets

# Training loop
for epoch in range(10):
    # Forward pass
    outputs = model(inputs)
    loss = criterion(outputs, targets)

    # Backward pass
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (epoch + 1) % 10 == 0:
        print(f"Epoch [{epoch+1}/10], Loss: {loss.item():.4f}")

[!IMPORTANT] By iteratively performing forward and backward passes over multiple epochs, the network learns the optimal parameters to minimize the loss and generalize to unseen data.

Importance of MLP

By combining these techniques, MLPs become powerful tools for modeling both linear and non-linear patterns in data. They form the foundation of many advanced deep learning architectures, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

June Journal

Anshul Raj Verma — Sat, 01 Jun 2024 00:00:00 GMT

Week 23 Journal

:partying_face: Found the solution of last point in Week 22 which occurs with global Taskfile.
- Just use USER_WORKING_DIR special variable of Taskfiles.
```
pc-all:
  desc: Run pre-commit on all files
  dir: "{{.USER_WORKING_DIR}}" # Refer to the absolute path of the directory `task` was called from.
  cmd: pre-commit run --all-files
```
- Related Docs:
Created HuggingFace account because now I want to dive into LLMs and want to create some basic application using them and in the flow of creating something I can learn DL and LLM concpets too.
:ballot_box: Created India Election 2024 dashboard using Streamlit. Used httpx.AyncClient, polars.LazyFrame and only async functions.
- Created the dashboard same day and posted on LinkedIn about it :star_struck:.
Add a reference link in sidebar to @arv-anshul/notebooks and @arv-anshul/dotfiles repo in /project page.
mkdocs-material/discussions, I want a feature through which I can replace the bullets of lists to a SVG icon.
- A guy suggested a way to achieve that but that is complicated and not much flexible.
@arv-anshul/yt-watch-history-v2
- :sparkles: Add Channel Recommender System.
- :recycle: Refactor the CTT ML Model.
A nice discussion with @MayankVanik around Gen AI, LLMs, AI Agents, LangChain, LangGraph and HuggingFace.
Just realize the best coding font for me which is Recursive Font. BTW, I've used it earlier but rejected but now when I realize the casual nature of it, I fell for it. Now, I am using its Rec Mono Casual (for markdowns & docs) and Rec Mono Dutone (for coding) font variants.

Week 24 Journal

I saw a guy in "We write code" Server by Hitesh Chaudhary who has amazing portfolio and he had done amazing work around Full Stack Development and he also contributed in multiple Open Source repos (big repos) :AMAZING:. He made a full-pledged WORKING clown of PW website and also made a desktop app of PW using Tauri (Rust). He has amazing portfolio website. There is so much things I can learn form him. @arnvgh
Completed the refactoring of CTT Model in @arv-anshul/yt-watch-history-v2 project. See release v0.3.0.
~~Created a community server on Discord and added many interactive features which helps others to connect and learn together as community.~~
- After creating the discord server, I've deleted it because I don't think that this is for me.
Wrote a clear project explanation paragraph of @arv-anshul/yt-watch-history-v2 project. See on website.
A employee from AABEE (a real state startup) reached me through reached me through my OG 99acres-scrape project. He conducted a meeting on G-Meet and given a task to scrape the older prices of properties to tackle the property's price trend but the data is not there on the website. That's why he said "we'll reach you later for any other work" :disappointed:.
The 3 co-founders of LeapX.ai startup reached me through my OG 99acres-scrape project. They conducted a meeting on MS Teams and the meeting goes for 48 Minutes while meeting they ask for my education, data science knowledge. In the end, they asked me a problem to explain (not solve; just explain) on the basis of explain they offer me a internship :star_struck:.
- I first interacted with Aditya, when he cam to know that I am in class 12th then started asking me about my Father and his profession and my place, how I study there, which school.
- One of them is from Makhdumpur :star_struck:.
In LeapX.ai interview I was explaining my @arv-anshul/yt-watch-history-v2 project and while explaining I mentioned the User Sentiment Analysis model which is able to predict the user's sentiment for programming, politics, entertainment, movie genre, etc. But the problem is that I haven't implemented it (in short I don't even checked it in ipynb). So, I decided to create a notebook around it and try to explain how to do it.
- :HARD: A ML Model which predict the user's sentiment around programming, politics, entertainment.
- :EASY: ~~A Clustering Model which cluster similar videos~~
- :EASY: A wordcloud which shows most common words around programming, politics, entertainment from user's watch history and by analysing that wordcloud anyone is able to determine their sentiment/habit.

Week 25 Journal

Project og-py to create custom GitHub repo social preview image using https://og-playground.vercel.app/ playground.
:star: Working at LeapX.ai as Intern.
:llama: Try using ollama in Zed editor. It is great but the response time very bad but I'll use it.
Read/Watch something about Playwright (better alternative of Selenium).
:handshake: Connected with @ujjwal-basnet and @iamrajharshit from Discord.

Internship at LeapX.ai

2024-06-19

After a one day off, We (Me and Aditya) connect on and discuss the affluent properties clustering problem. I have created some dashboard around it but after the meeting the conclusion is that data is not good because many properties were removed from 99acres's website. So we decided to scrape new data from the website. So I have to write code for that.

2024-06-20

While scraping data from 99acres with selenium I got many errors then I decided to scrape data from other website like Housing.com or MagicBricks.com.

I am planning to use selenium-wire. And, want to scrape more than one website and create a pipeline through which different websites' data combines together and forms a wholesome dataset to work on.

2024-06-21

Getting errors while using selenium-wire package, so I try to learn and use playwright. Again try to scrape using httpx with headers and cookies and I got success so I scrape almost 3K data using this. For now I'm sticking with this method.

Week 26 Journal

Thinking to create a GitHub Repo where all the scraping code for different websites are present. Also, repo maintains a README.md file for different scraping project, tools and resources.
Used st.Page (new multipage UI update from Streamlit) while building dashboards for LeapX.
:star: Got Offer from LeapX.ai for 3 month intership.
I had been writing many scraping scraping for many websites like Housing.com, MagicBricks.com, Naukri.com, etc. That's why thinking to publish them as blogs because they are not a full-pledged project instead they are just python using which we can scrape websites and there is no guarantee that script forever.

:scroll: I'll just explain the script and paste it there in the blog.
Added LeapX.ai Internship on LinkedIn.
Stared learning LangChain from YouTube after Aditya told me to-do so.
🇮🇳 WON T20I

Internship at LeapX.ai

2024-06-25

Meeting started after :clock8: 20:00 and end at :clock1030: 22:30. We worked on dataset scraped from Housing.com contains ~12K rows.

We took a different approach to cluster properties, used polars.Expr.qcut function to categories properties on the basis of "PRICE_PER_UNIT_AREA". After this created a dashboard around this (with Streamlit).

Now the main problem is to find a dense region and create a circle around it to mark that region with a label eg. Luxury, Value, Affordable, etc. And for this I have to write a function which do these findings and creating regions.

2024-06-26

No meeting conducted.

Using httpx library to scrape all 21K properties from Housing.com website of Gurgaon, faced many website downtime but eventually got things work.

Received offer leeter from LeapX.

2024-06-29

Completed the two page streamlit dashboard.

Told me to learn about AI stuffs like Langchain and Llama Index. Already found an awesome video on it.

Architecture for V2

Anshul Raj Verma — Sun, 19 May 2024 00:00:00 GMT

Application Interaction

Architecture of backend, ml and frontend.
How do backend, ml and frontend interacts?

<details> <summary>Code for Diagram</summary>

styleMode plain

Container 1 [icon: docker, color: blue] {
ML Models [icon: machine-learning] {
    Conetnt Type Predictor [icon: machine-learning]
    Channel Recommender System [icon: machine-learning]
}
ML Models API [icon: fastapi, color: green]
}

YouTube Data v3 API [icon: youtube, color: red]
Container 2 [icon: docker, color: blue] {
MongoDB [icon: mongodb, color: green] {
    Video Details [icon: database]
    Channel Details [icon: database]
}
Backend API [icon: fastapi, color: green]
}

Container 3 [icon: docker, color: blue] {
Frontend [icon: streamlit, color: red]
}

User [icon: user]

User > Container 3
Container 3 > Backend API, ML Models API
Backend API > MongoDB
Backend API > YouTube Data v3 API
ML Models API > ML Models

</details>

Frontend and Backend API Interaction

What happens when user upload their watch-history.json data to see analysis?
More and better data retrieval for analysis.
Use of YouTube Data v3 API to fetch more details of watched videos.

sequenceDiagram
    participant Frontend as Frontend
    participant API as API
    participant ChannelDetails as ChannelDetails
    participant VideoDetails as VideoDetails

    Frontend ->> API: Request all IDs from User's watch history

    Note over Frontend, API: User's watch history IDs

    activate Frontend
        API ->> ChannelDetails: Exclude IDs already available in DB
        ChannelDetails -->> API: IDs not in Database
        API -->> Frontend: Filtered IDs
    deactivate Frontend

    Note over Frontend, API: Filtered IDs

    Frontend ->> API: Fetch video details of filtered IDs

    Note over Frontend, API: Video details request

    activate API
        API ->> VideoDetails: Store fetched details
        VideoDetails -->> API: Stored!
        API ->> ChannelDetails: Store channel's videos ID from fetched data
        ChannelDetails -->> API: Stored!
        API -->> Frontend: Video Details
    deactivate API

    Note over Frontend: Show graphs using fetched data

<details> <summary>Code for Diagram</summary>

styleMode plain

Frontend [icon: streamlit, color: red]
API [icon: fastapi, color: green]
ChannelDetails [icon: mongodb, color: purple, label: Channel Videos ID DB]
VideoDeatils [icon: mongodb, color: orange, label: Video Details DB]
Frontend > API: All IDs from User's watch history

activate Frontend
API > ChannelDetails: Exclude IDs already available in DB
ChannelDetails > API: IDs not in Database
API > Frontend: Filtered IDs
deactivate Frontend

Frontend > API: Fetch video details of filtered IDs

activate API
API > VideoDeatils: Store fetched details
VideoDeatils > API : Stored!
API > ChannelDetails: Store channel's videos ID from fetched data
ChannelDetails > API : Stored!
API > Frontend: Video Details
deactivate API

</details>

ML Model Working

How does ML Model inference through API?
~~How to train ML Model on my custom data?~~

<details> <summary>Code for Diagram</summary>

direction right
styleMode plain

user [icon: user, shape: diamond, label: User]
inputData [icon: data, shape: cylinder, label: Input Data]
model [icon: machine-learning, shape: document, label: ML Model]
mlApi [icon: fastapi, label: ML API]
modelOnWeb [icon: globe, shape: oval, label: Model On Web]
prediction [icon: graph, label: Model Prediction]

user > inputData
inputData > mlApi
mlApi > model: Exists
mlApi > modelOnWeb: Not Exists
modelOnWeb > model: Download and Store
model > prediction
prediction > user

</details>

User Data Flow

Data manipulation after user uploads their data on application.
Use of YouTube Data v3 API to fetch more details about watched videos.
Leveraging the power of Python libraries like Polars for data manipulation. We can use Pandas also.

<details> <summary>Code for Diagram</summary>

styleMode plain

watchHistory [icon: user, color: blue] {
    title_ string
    titleUrl string
    time datetime
    subtitle dict
}

preprocessedWatchHistory [icon: pandas, color: blue] {
    videoId string
    title_ string
    channelId string
    channelTitle string
}

watchHistory.title_ - preprocessedWatchHistory.title_
watchHistory.titleUrl - preprocessedWatchHistory.videoId
watchHistory.subtitle - preprocessedWatchHistory.channelTitle
watchHistory.subtitle - preprocessedWatchHistory.channelId

videoDetails [icon: youtube, color: red] {
    id string
    title_ string
    channelId string
    description string
    tags list[string]
    publishedAt datetime
}

mergedData [icon: table, color: yellow] {
    videoId string
    title_ string
    channelId string
    channelTitle string
    description string
    tags string
    publishedAt datetime
}

preprocessedWatchHistory.videoId - mergedData.videoId
preprocessedWatchHistory.title_ - mergedData.title_
preprocessedWatchHistory.channelId - mergedData.channelId
preprocessedWatchHistory.channelTitle - mergedData.channelTitle
videoDetails.tags - mergedData.tags
videoDetails.description - mergedData.description
videoDetails.publishedAt - mergedData.publishedAt

</details>

<details> <summary>Diagram with Mermaid</summary>

erDiagram
    WATCH_HISTORY {
        title_ string
        titleUrl string
        time datetime
        subtitle dict
    }

    PREPROCESSED_WATCH_HISTORY {
        videoId string
        title_ string
        channelId string
        channelTitle string
    }

    VIDEO_DETAILS {
        id string
        title_ string
        channelId string
        description string
        tags list
        publishedAt datetime
    }

    MERGED_DATA {
        videoId string
        title_ string
        channelId string
        channelTitle string
        description string
        tags string
        publishedAt datetime
    }

    WATCH_HISTORY ||--|| PREPROCESSED_WATCH_HISTORY: title_
    WATCH_HISTORY ||--|| PREPROCESSED_WATCH_HISTORY: titleUrl
    WATCH_HISTORY ||--|| PREPROCESSED_WATCH_HISTORY: subtitle
    WATCH_HISTORY ||--|| PREPROCESSED_WATCH_HISTORY: subtitle

    PREPROCESSED_WATCH_HISTORY ||--|| MERGED_DATA: videoId
    PREPROCESSED_WATCH_HISTORY ||--|| MERGED_DATA: title_
    PREPROCESSED_WATCH_HISTORY ||--|| MERGED_DATA: channelId
    PREPROCESSED_WATCH_HISTORY ||--|| MERGED_DATA: channelTitle

    VIDEO_DETAILS ||--|| MERGED_DATA: tags
    VIDEO_DETAILS ||--|| MERGED_DATA: description
    VIDEO_DETAILS ||--|| MERGED_DATA: publishedAt

</details>

Resume Tips

Anshul Raj Verma — Thu, 09 May 2024 00:00:00 GMT

By Rohan Azad Sir

By implementing these strategies, you can elevate your Resume/CV to effectively communicate your qualifications and stand out among the competition in today's job market.

NOTE: I am writing this article from the POV of Data Scientists.

:memo: Important Points To Follow

Keep It Concise: Limit your CV to a single page to maintain reader interest and showcase your most relevant experiences succinctly.
Optimize Layout: Minimize white spaces and prioritize content placement for a visually appealing and easy-to-read document.
Highlight Relevant Projects: Tailor your CV to the job role by including specific projects that demonstrate your skills and expertise in alignment with the position.
Include Important Links: Provide links to your LinkedIn, GitHub, and Kaggle profiles to offer recruiters a deeper insight into your professional background and achievements.
Showcase Certifications: Feature 2 to 4 certifications that highlight your commitment to continuous learning and validate your proficiency in relevant areas.
Emphasize Technical Skills: Dedicate a section to showcasing your technical prowess, listing skills such as Statistics, GenAI, Power BI, and other job-specific technologies to demonstrate your capability to excel in the role.
Tweak CV For Job Role: Tweak your CV before submitting for job role, company's requirements; don't add unnecessary projects or tech skills other than required one.

:bookmark: How to list projects?

Project starts with Project Title.
Define project assessment in bullet points.
Must write the results (like RMSE, MAE of model). TLDR; Result has to be a number afterall.
Dedicate one or two points to define how you solve the problem using this project.

:art: Formatting

Result numbers must be in bold.
Use Sans Serif fonts like Calibri or Poppins.
Use font size between 10 to 12.
Too many formatting style must be restricted.

:pen: Writing

Check Grammatical errors. Use ChatBots to tackle this.
Use Action Verbs in your Resume. Checkout some well-known words below
🙅 Don't add summary which shows you that you wants to grow or you'll help the company to grow and something like this.
You can skip your Profile Summary or About Me section; if your CV got bigger. But try to keep it short and crisp.

<details> <summary>List of Action Verbs</summary>

</details>

Download My Resume

May Journal

Anshul Raj Verma — Wed, 01 May 2024 00:00:00 GMT

Week 18 Journal

Continue working on credit-modeling project with Sambhav.
- Sambhav wrote documentation of this project.
- Added the project documentation in repo too. The documentation PDF is saved after it converted into my website's page.
Start working on freeapi-py project. Also shared the project on Hitesh Chaudhary's Discord Server.
- Added /todo, /quote, /gh routes.
- Posted freeapi-py project on LinkedIn.
Rahul recommended an amazing YouTube channel @probabl_ai which goes deep inside the libraries (scikit-learn, pandas, polars, duckdb) code.
After attending the DSMP 2.0 session on "Project Based Interview Session" (by Rohan Azad), I knew many important rules and conventions regarging Interviews.
- Know yourself so that you can introduce yourself very well without jargons for approx. 1.5 minutes to 2 minutes.
- Know your projects because you have to explain it very well any process/concept of your project can be asked by interviewers. You've to explain your project for 30 to 40 minutes.
- Know the basics interviewers will try to trick you with them.
I woke up at 4 AM and immediately called my friends, Priyanshu, Suraj, Binam, and Pushpam. After a quick chat, we left home at 4:30 AM to play cricket on the PP School ground.

Week 19 Journal

Re-think md_badges.py CLI tool. Now it works very well.
curl -sSL url_to_file.py | python - arg1 arg2 arg3
LinkedIn post on ARC Browser also requested to download it with my referral link.
Opened a new discussion #2 in arv-anshul.github.io github repository.
Wrote a new article /ref/resume-tips.

Week 20 Journal

Restart the yt-watch-history project as yt-watch-history-v2.
- Thinking the whole process from scratch, draw some digrams on app.eraser.io.
- Copied the code of api/ and ml/ but writing the code for frontend from scratch.
Creating a website for Rahul using mkdocs-material. Check it out here.
:sparkles: Added recommendation pages for Music, Movies, Shows and Anime on the website.
- First, I've decided to maintain them here in diary but I got a solution which hide those page from website and at the same they are explicitly accessible from URL.
  
  <figure markdown>
  
  Page URL
  
  Music /music
  
  Movies & Shows /movies-and-shows
  
  Anime /anime
  
  </figure>
Disable, Dependendant Bot :robot: analysis on yt-watch-history repository.
Assign keyboard shortcut to raycast's emoji extension. Now pasting is very awesome :partying_face:.
Found the way to import my CSS and JavaScript files to add them in mkdocs.yaml.
- It helps to sync all the extra and custom CSS/JS files from main website where I maintin them.

Page	URL
Music	`/music`
Movies & Shows	`/movies-and-shows`
Anime	`/anime`

Changed battries of Mouse and Keyboard on 18 May, 2024.

mouse:
  old:
    name: Eveready
    price: ₹30
    installed_on: 2023-12-02
  new:
    name: Nippo
    price: ₹10
    installed_on: 2024-05-18
keyboard:
  old:
    name: Duracell
    price: included in box
    installed_on: on purchase date
  new:
    name: Nippo
    price: ₹10
    installed_on: 2024-05-18

Week 21 Journal

Updated the GitHub Profile README.
- Shorten the About Me section.
yt-watch-history-v2
- Created more diagrams.
- Now, download ML Model from URL (if not exists).
Custom Stylus Themes: I customize website's UI using stylus extension and for that I've to write CSS. So I'm going upload all those .user.css custom CSS in GitHub repo.
- Customized: youtube.com, gemini.google.com, chatgpt.com, open.spotify.com, linkedin.com and more.
- BTW, I also use ARC Boost for quick customization but I it doesn't give me ability to export the customization in CSS format.
This week update:
- Painting in my room took 2 Days.
- Installing Gypsum False Celling in my room disturb the daytime.
Get the current week number with this python command.

# using bash
echo "Current week number is $(date +%V)."

# using python
python -c "from datetime import date; print(f'Current week number is {date.today().isocalendar()[1]}.')"

Opened issues in zed, simpleicons, CampusX-Course repositories. Check my github activity.
Used mkdocs-redirects plugin in website repo. I found it very helpful and easy to use 😇.
A github repo which contains materials of CampusX Free Courses.

Week 22 Journal

New release for campusx repo (previously know as campusx-dsmp repo)
- See the release note for v0.11.0
Learned Taskfile (alternative to Makefile). Taskfile is better, easy, intuitive to use than Makefile.
- Pushed global Taskfile.yaml into dotfiles repository.
Marimo is a alternative to Jupyter Notebooks.
- I came to about it while juggling between Zed isuess.
- Also, Marimo has a blog where they discussed that how Marimo has overhead on Jupyter Notebooks.
After using Marimo:
- It has very functioning than Jupyter Notebooks because Marimo works on .py files instead of .ipynb files.
- It has better markdown support. You can natively pin the python vars in MDs.
- It has builtin support to convert a Jupyter notebook or Markdown file to a marimo script.
- You can use marimo but it has long learning curve.
- BTW, currently I uses VSCode builtin Notebooks but looking lokking forward to switch to Zed (when they supports it).
- Currently, I have no plans to switch to Marimo.

There is a big problem, I am facing with Taskfile.

Here, when you run Taskfile from different dir while referring to another Taskfile (present in different dir) with -t or -d flag, it will not take reference of pwd instead take reference of dir where the Taskfile present.

This is a big negative for me because I've defined a global Taskfile through which I ran many useful and important commands.

$ pwd
~/Developer/diary
$ task -t ~/Developer/dotfiles/Taskfile.yaml pc-all
task: [pc-all] pwd
~/Developer/dotfiles
task: [pc-all] pre-commit run --all-files
An error has occurred: InvalidConfigError:
=====> .pre-commit-config.yaml is not a file
Check the log at ~/.cache/pre-commit/pre-commit.log
task: Failed to run task "pc-all": exit status 1

Previously, I uses Makefile for this and there it works fine.

$ pwd
~/Developer/diary
$ make -f ~/Developer/Makefile pc-all
# works fine...

Credit Risk Modeling

Anshul Raj Verma — Tue, 30 Apr 2024 00:00:00 GMT

Problem Statement

Creating a machine learning model that can precisely segregate customers into class of giving credit based on past financial data and other pertinent borrower characteristics including income, credit score, and loan details is the aim. The likelihood that a borrower will fail on a loan should be estimated by the model, allowing it to determine the risk of lending to them. Such models can help financial institutions identify and measure their total risk exposure, set appropriate risk limits, and make informed investment decisions.

Project Workflow

Challenges Faced

One of the biggest challenges in credit risk modeling is the limited availability of relevant and reliable data. Credit risk models require historical data on loan performance, default rates, and economic indicators to accurately assess the likelihood of default.

Challenges include data availability, data quality, complex modeling, and regulatory compliance.

Example: One common challenge faced by financial institutions is obtaining accurate and reliable data for credit risk modeling purposes.

Detail Model Description

There are two datasets. We need to solve the challenges that are faced by the bank during credit lending.
First dataset is (i) Internal bank dataset and second dataset is (ii) Civil external dataset.
The target variable is Approved_Flag which contain 4 categories ['P2', 'P1', 'P3', 'P4'], segregating the customer into class of giving the credit. P1 being the category where the bank can easier give the credit to that customer whereas P4 being the category where it is not a good idea to give the credit to that customer, as it can increase the NPA accounts (Non-Performing assets) of the bank.
There are total 84 columns in two datasets. 26 columns in the first dataset and 62 columns in the second dataset.
PROSPECTID col is a common column in both the first and second datasets indicating unique customer ID.
To find association between numerical and numerical columns we will perform VIF test (Variance Inflation Factor). Reject columns whose p value is greater than a particular threshold.
For feature selection, we will perform Chi2-Square test and ANOVA test, since the target column is multi-class categorical column.
By checking the p_value of each column w.r.t. target variable, we can decide if it's statistically significant or not.
Made two models. One without credit score and another with credit score.
It is observed that the accuracy of model without credit score feature has dramatically decreases.
Without credit score the accuracy is 77% and with credit score the accuracy is 99%.

Dataset Columns Description

Bank Dataset

Column	Description
`pct_tl_open_L6M`	Percent accounts opened in last 6 months
`pct_tl_closed_L6M`	percent accounts closed in last 6 months
`Tot_TL_closed_L12M`	Total accounts closed in last 12 months
`pct_tl_closed_L12M`	percent accounts closed in last 12 months
`Tot_Missed_Pmnt`	Total missed Payments
`CC_TL`	Count of Credit card accounts
`Home_TL`	Count of Housing loan accounts
`PL_TL`	Count of Personal loan accounts
`Secured_TL`	Count of secured accounts
`Unsecured_TL`	Count of unsecured accounts
`Other_TL`	Count of other accounts
`Age_Oldest_TL`	Age of oldest opened account
`Age_Newest_TL`	Age of newest opened account

Civil Dataset

Column	Description
`time_since_recent_payment`	Time Since recent Payment made
`max_recent_level_of_deliq`	Maximum recent level of delinquency
`num_deliq_6_12mts`	Number of times delinquent between last 6 and last 12 months
`num_times_60p_dpd`	Number of times 60+ dpd
`num_std_12mts`	Number of standard Payments in last 12 months
`num_sub`	Number of sub standard payments - not making full payments
`num_sub_6mts`	Number of sub standard payments in last 6 months
`num_sub_12mts`	Number of sub standard payments in last 12 months
`num_dbt`	Number of doubtful payments
`num_lss`	Number of doubtful payments in last 12 months
`recent_level_of_deliq`	Number of loss accounts in last 12 months
`CC_enq_L12m`	Credit card enquiries in last 6 months
`PL_enq_L12m`	Personal Loan enquiries in last 6 months
`time_since_recent_enq`	Personal Loan enquiries in last 12 months
`enq_L3m`	Enquiries in last 6 months
`last_prod_enq2`	Lates product enquired for
`first_prod_enq2`	First product enquired for
`MARITALSTATUS`	Marital Status
`EDUCATION`	Education level
`AGE`	Age
`GENDER`	Sex
`Time_With_Curr_Empr`	Time with current Employer
`CC_Flag`	Credit card Flag
`PL_Flag`	Personal Loan Flag
`pct_PL_enq_L6m_of_ever`	Percent enquiries PL in last 6 months to last 6 months
`pct_CC_enq_L6m_of_ever`	Percent enquiries CC in last 6 months to last 6 months
`HL_Flag`	Housing Loan Flag
`GL_Flag`	Gold Loan Flag
`Approved_Flag`	Priority levels

Important Notes from both the Dataset

The shape of bank internal dataset of customer is (51336, 26).
The shape of civil dataset is (51336, 62)
The common column in both dataset is PROSPECTID which is unique ID for each customer.
The value "-99999" in both the datasets are null values.
We will remove all the null values if data lost is less than 20% of the total dataset.
Total trade lines is total no of accounts of a customer.

EDA

Unique Values in Categorical Columns

Column	Unique Values
MARITALSTATUS	Married, Single
EDUCATION	12TH, GRADUATE, SSC, POSTGRADUATE, UNDERGRADUATE, OTHERS, PROFESSIONAL
GENDER	M, F
last_prod_enq2	PL, ConsumerLoan, AL, CC, others, HL
first_prod_enq2	PL, ConsumerLoan, others, AL, HL, CC
Approved_Flag	P2, P1, P3, P4

After performing all the statistics tests (i.e. Chi-Square, VIF and ANOVA test), it is found that only 43 columns are important out of 82 columns.

[
  'Age_Newest_TL', 'Age_Oldest_TL', 'Approved_Flag', 'CC_enq_L12m', 'CC_Flag', 'CC_TL', 'EDUCATION',
  'first_prod_enq2', 'GENDER', 'GL_Flag', 'HL_Flag', 'Home_TL', 'last_prod_enq2', 'MARITALSTATUS',
  'max_recent_level_of_deliq', 'NETMONTHLYINCOME', 'num_dbt_12mts', 'num_dbt', 'num_deliq_6_12mts',
  'num_lss', 'num_std_12mts', 'num_sub_12mts', 'num_sub_6mts', 'num_sub', 'num_times_60p_dpd',
  'pct_CC_enq_L6m_of_ever', 'pct_PL_enq_L6m_of_ever', 'pct_tl_closed_L12M', 'pct_tl_closed_L6M',
  'pct_tl_open_L6M', 'PL_enq_L12m', 'PL_Flag', 'PL_TL', 'recent_level_of_deliq', 'Secured_TL',
  'time_since_recent_enq', 'time_since_recent_payment', 'Time_With_Curr_Empr', 'Tot_Missed_Pmnt',
  'Tot_TL_closed_L12M', 'Unsecured_TL', 'enq_L3m', 'Other_TL',
]

Data Visualization

Age Distribution Graph

The bank is majorly targeting people between 20 to 40.
The data distribution of this dataset is majorly spread between the age group of 20 to 40.

Age of Oldest Loan/Trade Line Account (in Months)

Time Since Recent Enquiry (in Months)

Credit Score Distribution

Most of the data distribution of credit score column is spread between 660 and 700 which fall under P2 category and that’s why majorly category in target column is P2 category only.

90 Percentile Monthly Income Distribution

This column illustrate that the salary income majority of people falls between 20k to 35k. It can be observed that the bank is mainly targeting those people whose is under 50k per month

Marital Status Distribution Graph

This column indicates that 73.1% of people who are applying for the loan are married.

Education Distribution Graph

The Graduate and 12th pass population contribute significantly to the dataset

Gender Distribution Graph

Gender wise, the dataset shows that 88.7% who are applying for loan are male or we can say that the bank is targeting male candidate more.

Last Product Enquiry Graph

First Product Enquiry Graph

Distribution of Target Variable Categories

60% of the people in the dataset falls under P2 category for loan approval.

Data Visualization - Observations

73% people are married people in the dataset.
This dataset contains have 88% of people as men who are likely to taken loan from the bank.
Graduate people have more likelihood of taking or applying for loans. Banks also have more likelihood of approving of loans to the graduate or educated people.
Previous loans taken by the people in this dataset is other loan or consumer loans (such as furniture loan, fridge loan etc).
Most of the people in the dataset flows under P2 category for loan approval.

Minimum, Maximum and Median value of `Credit_Score` for each category

The min and max credit score is P3 category is 489 and 776 respectively. This range indicates that for P3 category creates a big ambiguity for the model to predict the output accurately. For P1 and P2 categories, it is easier for the model to predict as it range from (701, 809) and (689 and 700) respectively.

Minimum, Maximum and median value of `AGE` for each category

Min age for all the category is 21. Max age varies from 63 to 67. It can be observed that for P1 category the median age is higher as compare to other categories and as the category decrease median age also decreses.

Maximum and median value of Age Oldest Trade Line accounts for each category

Maximum and median value of Time with current enquiry accounts for each category

Observation of numerical and categorical cols w.r.t target variable i.e. (Approved_Flag)

P1 category range is (701-809)
P2 category range is(669-700)
P3 category range is (489-776)
P3 category of target variable are the most ambiguous category. This can be observed by looking at the credit score min and max value for P3 category which range from 489 to 776, whereas in case of P2 it's ranges from 669 to 701.
Due to the most ambiguous category i.e. P3, during the predict also, the accuracy of the model is significantly decreases due to the most ambiguous category.
The median age who are getting P1 category loan are bit older than other categories. For eg. median age for P1 category is 40 whereas for P2 category it is 33 and for P3 category it is 31. Therefore it can be assumed that as the age increases, loan approval becomes easier.

Model Training

We used ensemble techniques (bagging and boosting) to train the model. Mainly we used RandomForestClassifier and XGBoostClassifier for classification. However, it is observed that XGBoost classifier has better accuracy as compare to RandomForest classifier. With accuracy 99% XGBoost is the best ML algorithm for the dataset with credit score feature is included. However, when credit score feature is excluded, there is a significant drop in the accuracy(76%) because the P3 category is most ambiguous category, the accuracy of the model is significantly decreases.

Classification Report Of `RandomForest`

Using `Credit_Score` Feature

	precision	recall	f1-score	support
P1	0.94	1.00	0.97	1224
P2	1.00	1.00	1.00	6397
P3	1.00	0.95	0.98	1595
P4	1.00	1.00	1.00	1309
accuracy			0.99	10525
macro avg	0.99	0.99	0.99	10525
weighted avg	0.99	0.99	0.99	10525

Without `Credit_Score` Feature

	precision	recall	f1-score	support
P1	0.82	0.70	0.75	1224
P2	0.79	0.93	0.86	6397
P3	0.45	0.21	0.28	1595
P4	0.75	0.70	0.72	1309
accuracy			0.77	10525
macro avg	0.70	0.63	0.65	10525
weighted avg	0.74	0.77	0.74	10525

Classification Report Of `XGBoost`

Class	P1	P2	P3	P4
Precision	0.813	0.826	0.434	0.772
Recall	0.788	0.912	0.305	0.698
F1 Score	0.800	0.866	0.358	0.733

Hyperparameter Tuning

Using skopt.BayesSearchCV, we got the best parameter for our XGBoost model. The parameters are:

{
  "alpha": 10,
  "colsample_bytree": 0.9,
  "learning_rate": 1.0,
  "max_depth": 3,
  "n_estimators": 100,
  "num_classes": 4,
  "objective": "multi:softmax",
}

With these parameters, accuracy increases by 1% when the model is trained without Credit_Score feature.

How To Perform EDA

Anshul Raj Verma — Mon, 22 Apr 2024 00:00:00 GMT

Performing EDA on a dataset is very difficult and time taking process because there is many thing you can do while performing EDA on your dataset.

I will generally use polars library in this article.

For plotting I'll generally use seaborn library.

First perform univariate analysis without any column dropping, after that perform BiVariate analysis and then decide whether to drop a feature or not. Also, provide a argument which proofs that your dropping decision is right.

Numerical Features

There are some basic Descriptive Analysis you can do with numerical column like mean, median, quantiles, etc. You can do this by calling the polars.DataFrame.describe method.

Univariate

Skewness

Skewness is a measure that describes the asymmetry or lack of symmetry in the distribution of a dataset. The value of skewness can provide insights into the shape and nature of the data distribution. Here's how to interpret skewness values:

Skewness Value	Interpretation
Negative Value	The distribution is left-skewed or negatively skewed. This means the tail on the left side of the distribution is longer, and the bulk of the values are concentrated on the right side of the distribution. The mean is typically less than the median.
Positive Value	The distribution is right-skewed or positively skewed. This means the tail on the right side of the distribution is longer, and the bulk of the values are concentrated on the left side of the distribution. The mean is typically greater than the median.
Zero	The distribution is symmetric, with no skew. The mean, median, and mode are approximately equal, and the distribution is evenly balanced on both sides.

The magnitude of the skewness value also provides information about the degree of skewness:

If the skewness value is close to 0 (between -0.5 and 0.5), the distribution is approximately symmetric with little to no skew.
If the skewness value is significantly negative (below -1), it suggests strong left skewness.
If the skewness value is significantly positive (above 1), it suggests strong right skewness.

In summary, the interpretation of skewness values is as follows:

Negative skewness: The distribution is left-skewed, with the tail extending towards the left.
Positive skewness: The distribution is right-skewed, with the tail extending towards the right.
Zero skewness: The distribution is symmetric, with no skew.
Magnitude of skewness: Indicates the degree of skewness, with values close to 0 suggesting little to no skew, and values below -1 or above 1 suggesting strong skewness.

Use polars.Series.skew method to calculate skewness of a numerical feature.

Kurtosis

Kurtosis is a statistical measure that describes the degree of peakedness and tailedness of a probability distribution. It provides information about the shape of the distribution, specifically the proportion of data that is concentrated in the tails compared to the normal distribution.

There are three main types of kurtosis:

Mesokurtic: A distribution with a kurtosis value of 3, which is the same as a normal distribution. This indicates a moderate level of peakedness and tailedness.
Leptokurtic: A distribution with a kurtosis value greater than 3. This indicates a higher, more peaked distribution with heavier, fatter tails compared to a normal distribution. Leptokurtic distributions have more outliers and extreme values.
Platykurtic: A distribution with a kurtosis value less than 3. This indicates a flatter, more dispersed distribution with lighter, thinner tails compared to a normal distribution. Platykurtic distributions have fewer outliers and extreme values.

The excess kurtosis is calculated by subtracting 3 from the kurtosis value. Positive excess kurtosis indicates a leptokurtic distribution, while negative excess kurtosis indicates a platykurtic distribution.

In summary, kurtosis provides information about the shape of a distribution, specifically the concentration of values in the tails compared to a normal distribution. Higher kurtosis indicates more outliers and extreme values, while lower kurtosis indicates fewer outliers and a more dispersed distribution.

Use polars.Series.kurtosis method to calculate kurtosis value of a numerical feature. This method has fisher argument, if fisher=True then normal is 0.0; if fisher=False then normal is 3.0.

Distribution

kdeplot: Plot distribution of data using Kernel Density Estimation.
histplot: Plot histogram of data.
ecdfplot: Plot empirical cumulative distribution functions.
rugplot: Plot marginal distributions by drawing ticks along the x and y axes.

See seaborn library docs to plot different data distribution.

Box Plot

Plot the 5-number summary of data with BoxPlot.

What is 5-number summary of a data?

This represents the #!math (\text{Q1} - 1.5*\text{IQR}), 25th %tile #!math (\text{Q1}), 50th %tile, 75th %tile #!math (\text{Q3}) and #!math (\text{Q3} + 1.5*\text{IQR}) values of data.

BoxPlot easily shows the outliers of the data.

Percentile

You can use mean or median to know the central value of data. But what about identifying specific data points relative to the entire dataset's distribution?

Percentiles offer a solution by dividing the data into hundredths and determining where a particular value falls within that range. They provide valuable insights into the spread and distribution of data, aiding in comparisons and understanding the dataset's overall characteristics.

You can use percentile to know dataset's outlier values. After ploting BoxPlot you can manually check the outlier values with 95%tile or 99%tile.

Use polars.Series.quantile method to calculate nth quantile of dataset.

BiVariate

Regression Plots

Regression Plot: Scatter plot but with a best-fit regression line in there.
Residual Plot: Plot the residuals/error of a regression model and check whether it any linear relationship.

See seaborn library docs for regression plots details.

Relational Plots

Scatter Plot: Check the linear relationship between two datasets.
Line Plot: Plot the relationship between x and y with many parameters such as hue, size and style.

See seaborn library docs for relational plots details.

More Plots

FacetGrid
PairPlot
JointPlot

Checkout seaborn docs for more details.

MultiVariate

Correlation

MultiCollinearity

Extra Concepts

Homoscedasity (concept)

Categorical Features

Univariate

CountPlot

BiVariate

Contengency Table

MultiVariate

Correlation

Hypothesis Testing

Numerical VS Numerical

Z-test

T-test

Categorical VS Categorical

Chi-Square

Numerical VS Categorical

OneWay ANOVA

TwoWay ANOVA

Basics of Statistics for ML

Anshul Raj Verma — Fri, 12 Apr 2024 00:00:00 GMT

graph TD;
    Inferential_Statistics["Inferential Statistics"]
    Descriptive_Statistics["Descriptive Statistics"]
    Measure_of_Central_Tendency["Measure of Central Tendency"]
    Weighted_Mean["Weighted Mean"]
    Trimmed_Mean["Trimmed Mean"]
    Measure_of_Dispersion["Measure of Dispersion"]
    Standard_Deviation["Standard Deviation"]
    CV["Coefficient of Variation"]
    Five_Number_Summary["5 Number Summary"]
    Box_Plot["Box Plot / Whisker Plot"]

    Statistics --> Descriptive_Statistics
    Statistics --> Inferential_Statistics
    Descriptive_Statistics --> Measure_of_Central_Tendency
    Descriptive_Statistics --> Measure_of_Dispersion
    Measure_of_Central_Tendency --> Mean
    Measure_of_Central_Tendency --> Median
    Measure_of_Central_Tendency --> Mode
    Mean --> Weighted_Mean
    Mean --> Trimmed_Mean
    Measure_of_Dispersion --> UniVariate
    Measure_of_Dispersion --> BiVariate
    UniVariate --> Range
    UniVariate --> Variance
    UniVariate --> Standard_Deviation
    UniVariate --> CV
    UniVariate --> Five_Number_Summary
    Five_Number_Summary --> Percentile
    Five_Number_Summary --> Box_Plot
    BiVariate --> Covariance
    BiVariate --> Correlation

Statistics

Population

Population represents the whole/entire group of individual or object that we are interested in studying.

Sample

Sample is a subset of Population. It is smaller group of individual or object that we select from the population to study. Samples are used to estimate characteristics of the population.

[!TIP] Things to care while selecting Sample from a Population

Sample Size

Random Data

Representative

[!IMPORTANT] Example: Population and Sample Data

All Python Programmer on the earth VS Python Programmer in India

All cricket fans VS Fans who were present in the stadium

All students VS Who visit college for lectures

Descriptive Statistics

Helps you to learn underlying information of the data. Generally, data is a Sample of the Population.

The goal of statistical inference is to use the information obtained from the sample to make inferences about the population parameters.

Measure of Central Tendency

A measure of central tendency is a statistical measure that represents a typical or central value for a dataset. It provides a summary of the data by identifying a single value that is most representative of the dataset as a whole.

You can use methods like Mean, Median, Mode for this.

Mean: The mean is the sum of all values in the dataset divided by the number of values.
Median: The median is the middle value in the dataset when the data is arranged in order.
Mode: The mode is the value that appears most frequently in the dataset.

Measure of Dispersion

A measure of dispersion is a statistical measure that describes the spread or variability of a dataset. It provides information about how the data is distributed around the central tendency (mean, median or mode) of the dataset.

Variance

The variance is the average of the squared differences between each data point and the mean. It measures the average distance of each data point from the mean and is useful in comparing the dispersion of datasets with different means.

Sample Variance	Population Variance
`#!math \sigma^2 = \frac{\sum\_{i=1}^{n} (x_i - \bar{x})^2}{n-1}`	`#!math \sigma^2 = \frac{\sum\_{i=1}^{N} (x_i - \mu)^2}{N}`

[!IMPORTANT] Why n-1 in Sample Variance?

By dividing by #!math (n−1), we make the sample variance an unbiased estimator of the population variance. This correction is particularly important when dealing with small sample sizes, as it helps to reduce bias in the estimation of the population variance.

Standard Deviation

The standard deviation is the square root of the variance. It is a widely used measure of dispersion that is useful in describing the shape of a distribution like Normal Distribution.

Sample Standard Deviation	Population Standard Deviation
`#!math s = \sqrt{\frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n-1}}`	`#!math \sigma = \sqrt{\frac{\sum_{i=1}^{N} (x_i - \mu)^2}{N}}`

Coefficient of Variation (CV)

CV is the ratio of the standard deviation to the mean expressed as a percentage. It is used to compare the variability of datasets with mean. CV is a statistical measure that expresses the amount of variability in a dataset relative to the mean.

`#!math \frac{\sigma}{\mu} \cdot 100 = \text{CV}\%`

Why Coefficient of Variation Exits?

This method is used to compare the variance of two different feature like Salary of Employees and Experience of Employees. But using CV you can compare the variability of those two feature.

Steps

Calculate Mean and Standard Deviation of both feature.
Calculate CV of each feature using above formula.
Now, you can compare both feature's CV to know which is feature has more variability.

[!NOTE] This statistical method is not widely used.

Inferential Statistics

Using Inferential Statistics we make conclusions and predictions about a population based on a sample.

It involves the use of probability theory to estimate the likelihood of certain events occurring, hypothesis testing to determine if a certain claim about a population is supported by the data, and regression analysis to examine the relationships between variables.

Covariance

Covariance is a statistical measure that describes the degree to which two variables are linearly related. It measures how much those two variables change together, such that when one variable increases, does the other variable also increase, or does it decrease?

`#!math \text{Cov}(x, y) = \frac{\sum_{i = 0}^{n}{(x_i - \bar{x}) (y_i - \bar{y})}}{n - 1}`

Interpretation

If the covariance between two variables is positive, it means that the variables tend to move together in the same direction. If the covariance is negative, it means that the variables tend to move in opposite directions. A covariance of zero indicates that the variables are not linearly related.

Disadvantage

One limitation of covariance is that it does not tell us about the strength of the relationship between two variables, since the magnitude of covariance is affected by the scale of the variables.

Variance V/S CoVariance

Covariance is a measure of how much those two variables change together, while variance is a measure of how much a single variable changes from its mean.
The variance of a variable is the average of the squared differences from the mean, while the covariance between two variables is the average product of their deviations from their respective means.

Corelation

Correlation refers to a statistical relationship between two or more variables. Specifically, it measures the degree to which two variables are related and how they tend to change together.

`#!math \text{Correlation} = \frac{\text{Cov}(x, y)}{\sigma x * \sigma y}`

Interpretation

Correlation is often measured using a statistical tool called the correlation coefficient, which ranges from -1 to 1, A correlation coefficient of -1 indicates a perfect negative correlation, a correlation coefficient of 0 indicates no correlation, and a correlation coefficient of 1 indicates a perfect positive correlation.

Correlation and Causation

The phrase "correlation does not imply causation"[^1] means that just because two variables are associated with each other, it does not necessarily mean that one causes the other.

In other words, a correlation between two variables does not necessarily imply that one variable is the reason for the other variable's behavior.

[^1]: "correlation does not imply causation": Agar koi ghatna correlated hai toh iska matlab yeh nhi ki pehle ghatna ki hone ki wajah se he dusri ghatna ho rhi hai.

Suppose there is a positive correlation between the number of :firefighter: firefighters present at a fire and the amount of :adhesive_bandage: damage caused by the :fire: fire. One might be tempted to conclude that the presence of :firefighter: firefighters causes more :adhesive_bandage: damage. However, this correlation could be explained by a third variable - the severity of the fire. Severer fires might require more firefighters to be present, and cause more damage.

Thus, while correlations can provide valuable insights into how different variables are related, they cannot be used to establish causality. Establishing causality often requires additional evidence such as experiments, randomized controlled trials, or well-designed observational studies.

Bash Useful Commands

Anshul Raj Verma — Wed, 10 Apr 2024 00:00:00 GMT

Bash

Display ANSI Colors with Their Color-Code

function colormap() {
    range_start=${1:-1}
    range_end=${2:-255}

    for i in $(seq $range_start $range_end); do
        echo -en "\e[48;5;${i}m  ${(l:3::0:)i}  \e[0m "
        [[ $((i % 10)) -eq 0 ]] && echo
    done
    return 0
}

The colormap function will print the ANSI colors with codes in your terminal.

colormap           # (000 - 255)
colormap 200       # (200 - 255)
colormap 100 120   # (100 - 120)

Docker

Clear All Caches, Images, Containers, Volumes of Docker

docker system prune -a --volumes --force

Git

Beautiful One Line - `git log`

alias glog="git log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --branches"

$ glog
* 6914e04 - (HEAD -> main, origin/main) 📝 Update docs/index.md (5 hours ago) <Anshul Raj Verma>
*   3d91241 - Merge branch 'new/friends-page' (5 hours ago) <Anshul Raj Verma>
|\
| * 6160ae5 - 💅 Update UI of friends.md (5 hours ago) <Anshul Raj Verma>
| * 42b52d8 - 🔨 Add friends.md to Home nav (6 hours ago) <Anshul Raj Verma>
| * bef45f6 - ✨ Setup to add friends on website (2 days ago) <Anshul Raj Verma>
|/
* 59340cf - ref: ✨ Add starship.md in Bash (2 days ago) <Anshul Raj Verma>
* ecc8e94 - ✨ Add new nav item "References" (3 days ago) <Anshul Raj Verma>
* 3bea521 - ✨ Add Rust icon on about page (5 days ago) <Anshul Raj Verma>

Python

Remove All Cache File Specify to a Python Project

function pycls() {
    find . -name '.DS_Store' -type f -delete
    find . -type d -name "__pycache__" -exec rm -rfv {} \; 2>/dev/null
    find . -type d -name ".ipynb_checkpoints" -exec rm -rfv {} \; 2>/dev/null
    find . -type d -name ".ruff_cache" -exec rm -rfv {} \; 2>/dev/null
    find . -type f -name "*.pyc" -exec rm -fv {} \; 2>/dev/null
}

$ pwd
/User/Home  # For MacOS
$ pycls  # Remove all cache files and folders in the directory and sub-dirs

Homebrew

Update, Clean, and Doctor

brew update && brew cleanup --prune=all && brew doctor

Uninstall Unused Dependencies

brew autoremove

Check docs for autoremove command.

Arc

Change `ARC.app` Icon

Source Gist

defaults write company.thebrowser.Browser currentAppIconName candy # favorite

Candy Arc	Hologram	Neon	Fluted Glass	Schoolbook	Colorful
`candy`	`hologram`	`neon`	`flutedGlass`	`schoolbook`	`colorful`

Friends on Website

Anshul Raj Verma — Mon, 08 Apr 2024 11:02:00 GMT

I decided to create a page where I put my friends information like their name, description, and social links which helps others (who visit my website) to connect with them.

Planning

Define a JSON schema for friends.yaml file. See the json-schema.
Use builtin Grids to showcase the friend's information. See some experimental layouts.
Friends' description must be between 50-70 words. If they fail to maintain them then use ChatGPT to reduce the size.

How Do each Friend's Section Looks?

More certainly they look similar to one another because the web pages is being gets rendered from a yaml file to maintain the certainty.

[!question] Why render from yaml format?

Because I am going to define a schema for that which follows a certain format and using that format the webpage is being rendered using mkdocs-markdownextradata-plugin.

How to Add Friends?

I can manually asks their infos and add into yaml files.

I can open a discussion where my friends provide their infos and request me to add them on the web page.

My friends can open a PULL REQUEST into the repo which will automatically add them on webpage.

Layouts

These are my rough ideas which I prefer to replicate on my website's page to display the friends' information.

:material-numeric-1-box:{ .lg } Layout 1

{ .twemoji .xxl .middle .round } Anshul Raj Verma

<p align=justify>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tempus urna et pharetra pharetra massa massa ultricies mi. Adipiscing enim eu turpis egestas. Pretium fusce id velit ut tortor pretium viverra suspendisse potenti. Egestas egestas fringilla phasellus faucibus scelerisque eleifend. Sapien pellentesque habitant morbi tristique senectus et. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget.
</p>

<p align=center>
[:simple-github:{ .lg }](https://linkedin.com/in/arv-anshul) &nbsp;
[:simple-linkedin:{ .lg }](https://github.com/arv-anshul) &nbsp;
</p>

</div>

<p align=center>{ .twemoji .xxl .middle .round } Anshul Raj Verma</p>

<p align=justify>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tempus urna et pharetra pharetra massa massa ultricies mi. Adipiscing enim eu turpis egestas. Pretium fusce id velit ut tortor pretium viverra suspendisse potenti. Egestas egestas fringilla phasellus faucibus scelerisque eleifend. Sapien pellentesque habitant morbi tristique senectus et. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget.
</p>

<p align=center>
[:simple-github:{ .lg }](https://linkedin.com/in/arv-anshul) &nbsp;
[:simple-linkedin:{ .lg }](https://github.com/arv-anshul) &nbsp;
</p>

</div>

:material-numeric-2-box:{ .lg } Layout 2

{ .twemoji .xxl .middle .round } Anshul Raj Verma

<p align=justify>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tempus urna et pharetra pharetra massa massa ultricies mi. Adipiscing enim eu turpis egestas. Pretium fusce id velit ut tortor pretium viverra suspendisse potenti. Egestas egestas fringilla phasellus faucibus scelerisque eleifend. Sapien pellentesque habitant morbi tristique senectus et. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget.
</p>

<p align=center>
[:simple-github:{ .lg }](https://linkedin.com/in/arv-anshul) &nbsp;
[:simple-linkedin:{ .lg }](https://github.com/arv-anshul) &nbsp;
</p>

<p align=center>{ .twemoji .xxl .middle .round } Anshul Raj Verma</p>

<p align=justify>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tempus urna et pharetra pharetra massa massa ultricies mi. Adipiscing enim eu turpis egestas. Pretium fusce id velit ut tortor pretium viverra suspendisse potenti. Egestas egestas fringilla phasellus faucibus scelerisque eleifend. Sapien pellentesque habitant morbi tristique senectus et. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget.
</p>

<p align=center>
[:simple-github:{ .lg }](https://linkedin.com/in/arv-anshul) &nbsp;
[:simple-linkedin:{ .lg }](https://github.com/arv-anshul) &nbsp;
</p>

</div>

:material-numeric-3-box:{ .lg } Layout 3

<p align=center>{ .twemoji .xxxl .round }</p>

<p align=center>Anshul Raj Verma</p>

<p align=center> :simple-github:{ .lg } :simple-linkedin:{ .lg } </p>

<p align=justify>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tempus urna et pharetra pharetra massa massa ultricies mi. Adipiscing enim eu turpis egestas. Pretium fusce id velit ut tortor pretium viverra suspendisse potenti. Egestas egestas fringilla phasellus faucibus scelerisque eleifend. Sapien pellentesque habitant morbi tristique senectus et. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget.
</p>

<p align=center>{ .twemoji .xxxl .round }</p>

<p align=center>Anshul Raj Verma</p>

<p align=center>
[:simple-github:{ .lg }](https://linkedin.com/in/arv-anshul) &nbsp;
[:simple-linkedin:{ .lg }](https://github.com/arv-anshul) &nbsp;
</p>

<p align=justify>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tempus urna et pharetra pharetra massa massa ultricies mi. Adipiscing enim eu turpis egestas. Pretium fusce id velit ut tortor pretium viverra suspendisse potenti. Egestas egestas fringilla phasellus faucibus scelerisque eleifend. Sapien pellentesque habitant morbi tristique senectus et. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget.
</p>

<p align=center>{ .twemoji .xxxl .round }</p>

<p align=center>Anshul Raj Verma</p>

<p align=justify> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tempus urna et pharetra pharetra massa massa ultricies mi. Adipiscing enim eu turpis egestas. Pretium fusce id velit ut tortor pretium viverra suspendisse potenti. Egestas egestas fringilla phasellus faucibus scelerisque eleifend. Sapien pellentesque habitant morbi tristique senectus et. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget. </p>

<p align=center>
[:simple-github:{ .lg }](https://linkedin.com/in/arv-anshul) &nbsp;
[:simple-linkedin:{ .lg }](https://github.com/arv-anshul) &nbsp;
</p>

</div>

Bash Tips

Anshul Raj Verma — Sun, 07 Apr 2024 00:00:00 GMT

This page contains some Tips and Tricks which are helpful to work with Bash scripts or Terminals. I have mentioned only those tips and tricks which I uses most in my daily routine, BTW you can refer to links which I have shared to know all the tips and tricks.

Tips By `asottile`

I will only mention those commands and shortcuts which helpful for me. You can check all of them using below links.

Command: `!!`

A substitution which contains the previous command, some useful invocations.

$ ls /proc/1/exe
ls: cannot access '/proc/1/exe': Permission denied

$ sudo !!
sudo ls /proc/1/exe
/proc/1/exe

$ ls /tmp/ | grep sys
systemd-private-e21be514f23449189063b6bd95ec13ef-bolt.service-MCMIaN
systemd-private-e21be514f23449189063b6bd95ec13ef-colord.service-yNWr06
systemd-private-e21be514f23449189063b6bd95ec13ef-fwupd.service-geozA1

$ watch "!!"
watch "ls /tmp/ | grep sys"
(opens up watch panel)

Command: `!$`

A substitution which contains the last segment of the previous command.

$ $EDITOR test.py

$ python !$
python test.py
Hello world!

mkdir new-project
code !$  # Opens the `new-project folder in VSCode

Shortcut: <kbd>ctrl</kbd> + <kbd>\</Kbd>

<kbd>ctrl</kbd> + <kbd>\</kbd> sends SIGQUIT (default behavior: terminate + produce a core dump) which can be useful to kill things that normally catch <kbd>ctrl</kbd> + <kbd>c</kbd> (SIGINT).

:fire: <kbd>ctrl</kbd> + <kbd>\</kbd> is more powerful than <kbd>ctrl</kbd> + <kbd>c</kbd>.

Additional Tricks

Trick: cmd `!!`

Run previous command with new command.
Replace previous command with `!!` argument.

$ brew --prefix
/opt/homebrew

$ open `!!`
open `brew --prefix`
# Now Finder.app opens at "/opt/homebrew" path

Uninstall Apps from Mac

Uninstall and remove all app data from your Mac using this script. For more information see this youtube video.

Remove Mac's "Login Items"

Go to /Library/LaunchAgents and /Library/LaunchDaemons path.
Check for the login item names and delete them.
This process might asks for password.

Hey Wcowin

Anshul Raj Verma — Fri, 05 Apr 2024 16:23:00 GMT

I had opened a discussion in mkdocs-material GitHub repo and tomorrow and @Wcowin wrote a comment on that discussion and I replied him generously. After my reply he followed me and this makes me curious; "Why/How any random guy can follow me?" Because I can't do this to anybody.

Then, I started exploring his GitHub account and visited his personal website and that website is just wonderful place because that palace contains tons of thoughts :star_struck: of him.

[!NOTE] Wonderful

How can a person write ^^these types of thoughts in his diary/journal and made it public (I am not hateful, just surprised :exploding_head:), because I also get similar thought but can't able to maintain it like this. "Just wonderful :people_hugging:"

Link to page I don't know is he really thought this or just copied from somewhere.

Link to page He maintain his experiences of watching Dramas, Movies and TV Shows :nerd_face:.

Link to page WTF (Why The Fuck?) are these in public :face_with_peeking_eye:.

Link to page I haven't read this page bu it seems good; I am going to read this page soon.

I am thankful that I met you and thankful to you that you shared your thoughts like this and I'm able to read it (after translation). I am thankful to you that you met me and also looking forward to your next thoughts.

Meet you at your next thought.

On 1st May, 2024

I am searching his Friends page on his website (but navigating with Chinese language) but into his Blog page and after scrolling a bit I read my name (Anshul Raj Verma) because others words were written in Chinese which got my focus to my name. After sudden realization I copied the sentence and pasted it on Google Translate and came to know that he wrote:

[!IMPORTANT] Translation

In Chinese: 感谢Anshul Raj Verma在Discussions#7上提出的网站优化建议

In Hindi: चर्चा#7 में वेबसाइट अनुकूलन सुझावों के लिए अंशुल राज वर्मा को धन्यवाद

In English: Thanks to [Anshul Raj Verma] for his website optimization suggestions in [Discussions#7]

The Rust Language

Anshul Raj Verma — Tue, 02 Apr 2024 22:41:00 GMT

Finally, learning my ~~dream language~~ (after using some top-notch projects written in it) Rust.

Created new arv-anshul/thrust GitHub repo which contains my learning resources and my practice progams. I have written some thoughts in the repo's README.md file.

Thoughts

The Dream Language - Nightmare

Before learning Rust I imagine that this is my Dream Language for sure but after digging into it I regret. Why I started learning this. Because this is ***king hard too learn if you compare to Python (After this I think python has reduced the complexities to 50-60% :pray:).

The Learning Curve

My first language is Python and it very to learn because there is no concept of Ownership and 69+ different data-types; just write your code and run it. But in Rust the compiler says aise kaise.

The Rust Compiler

Mera koti-koti naman unko jo iis compiler (rustc) ka chintan kiye aur ise itna safal banaya. Matlab rustc ko sab pata hai ki aap kisko kya de rhe ho, kaise de rhe ho, sab kuch; inse kuch nhi chhipa hai. Agar kuch bhi idhar-udhar hua, turant tok denga.

Irritation

In the beginning, while creating programs or project it feels irritating; you have no idea how to do it there is so much constraints to care about. Sometimes I feels that I can build this in Python and that is enough for me. :angry:.

April Journal

Anshul Raj Verma — Mon, 01 Apr 2024 00:00:00 GMT

Week 14 Journal

Learning

Rust
- Learning Rust from YouTube and Rust Books.
- Created arv-anshul/thrust repo where I practice Rust by creating programs.
- End of Week: I decided to leave Rust behind and starts focusing on ML and DL. :wink:

Thoughts

Rust
- :tada: Finally, finally, finally.
- Read my thoughts on rust.
CampusX - Short Courses
- Just great to know that Nitish Sir is launching approx. 10 New Short Courses where other teachers will teaches some concepts in deep and also build projects using those concepts.
- Also, these short courses is free and for lifetime.
- I think it's just like short courses by Deeplearning.ai.
hey-wcowin.md
- I met @Wcowin and written something about him.

Week 15 Journal

Learning

mkdocs - nav
- Create multiple references of same notes in mkdocs.yaml's nav field and then it gets render as it is written. Checkout the official documentation.
- Also checkout website's mkdocs.yaml file's nav field.
Anshul Raj Verma - Friends
- Created a Friends page on website using where I'll put my friends info.
- The info were rendered from yaml file using mkdocs-markdownextradata-plugin.

Thoughts

mkdocs-edges
- Planning to write a plugin for mkdocs using which users are able to customize their sites layout (rounded edges).
Fresher Getting 17 LPA Data Scientist Job From Tier 3 College | Story of Tarun Chauhan
- Connect with Tarun on LinkedIn
- After watching this video; I have to make better notes from CampusX videos which helps me to remember those concepts workings and theory.

Week 16 Journal

Change ARC Browser Icon - GitHub Gist
Basics of Statistics for ML: New blog to learn basic statistics for Machie Learning. Explained Descriptive and Inferential Statistics.
Username Dilemma
- I am changing my username from (arv-anshul/arv_anshul) to (batook/batookm) because it is of one word, no separation (-/_) in between.
- I have already changed username of Telegram, Discord and Instagram.
  
  <figure markdown>
  
  Platform ID
  
  Discord batookm
  
  Telegram batookm
  
  Instagram _batookm
  
  </figure>
- ~~:thinking: Thinking to change GitHub username too but that is too overwhelming.~~
- I am not going to change GitHub and LinkedIn username because arv-anshul is my professional username and batook is my nick-username. If I made my X (formally Twitter) then I'll keep my username as batook :sunglasses:.
freeapi.app - By Hitesh Choudhary :sunglasses:
- I am thinking to recreate this in Python using FastAPI as freeapi-py.
- I first need to tackle:
  - How to handle limit query?
  - How to use MongoDB from localhost and Docker?
ruff/server
- Yet another banger from Astral devs.
- Read blog on ruff server.
Me and Rahul Bhaiya are started working on that yt-watch-history project but from scratch.
Write about "How to introduce yourself?".
Maintain a page where you write your achievement also express your thoughts on them.

Platform	ID
Discord	`batookm`
Telegram	`batookm`
Instagram	`_batookm`

Week 17 Journal

Discuss lots of questions with @Rahul-Kumar30.
Learned Confusion Matrix from Credit Risk Modeling Course.
Did Credit Risk Modeling project with @SMBHV.
Created a new Journal page anime.md where I put my experiences of watching anime.
Updated the convention of Journal. See
SOMTO gave his information to add it on website's friends page.
Binam asked for roadmap of Data Science.

Introduction To Hypothesis Testing

Anshul Raj Verma — Mon, 25 Mar 2024 00:00:00 GMT

Are you a Null Hypothesis or an Alternate Hypothesis? What does this question means?

Null V/S Alternative Hypothesis

Parameter	Null Hypothesis	Alternative Hypothesis
Definition	A null hypothesis is a statement in which there is no relation between the two variables.	An alternative hypothesis is a statement in which there is some statistical relationship between the two variables.
What is it?	Generally, researchers try to reject or disprove it.	Researchers try to accept or prove it.
Testing Process	Indirect and Implicit	Direct and Explicit
p-value	Null hypothesis is rejected if the p-value is less than the alpha-value; otherwise, it is accepted.	An alternative hypothesis is accepted if the p-value is less than the alpha-value otherwise, it is rejected.
Notation	`#!math H_0`	`#!math H_1`
Symbol	Used Equality Symbol (=, ≥, ≤)	Inequality Symbol (≠, <, >)

Null hypothesis assumes no effect or relationship, while the alternative hypothesis suggests the presence of an effect or relationship.
Researchers try to reject or disprove the null hypothesis, whereas they aim to accept or prove the alternative hypothesis.
Null hypothesis testing is indirect and implicit, while alternative hypothesis testing is direct and explicit.
Null hypothesis is represented by H0 with an equal sign, while the alternative hypothesis is denoted by H1 with an unequal sign.

:art: Illustrate Hypothesis Testing

Scenario

A researcher believes that a new teaching method improves student performance in mathematics compared to the traditional method. The average score of students using the new method is expected to be higher than the average score of students using the traditional method.

Hypotheses

Null Hypothesis #!math (H_0): The average score of students using the new teaching method is the same as or lower than the average score of students using the traditional method #!math (H_0:\mu_{\text{new}} \le \mu_{\text{traditional}}).

Alternative Hypothesis #!math (H_a): The average score of students using the new teaching method is higher than the average score of students using the traditional method #!math (H_a:\mu_{\text{new}} \gt \mu_{\text{traditional}}).

Steps

Formulate Hypotheses: Establish the null and alternative hypotheses.
Collect Data: Administer both teaching methods to two groups of students and collect their test scores.
Calculate Test Statistic: Compute the test statistic, such as the t-test, based on the sample data.
Compare to Critical Region: Compare the test statistic to critical values to determine if it falls in the critical region.
Calculate p-value: Calculate the p-value associated with the test statistic.
Make Conclusion: Based on the p-value and significance level, either reject the null hypothesis if the p-value is less than the significance level or fail to reject it.

In this example, ^^if the p-value is less than the chosen significance level (e.g., 0.05), you would reject the null hypothesis and conclude that there is evidence to support the claim that the new teaching method leads to higher student performance in mathematics^^. This demonstrates how hypothesis testing allows researchers to draw conclusions based on statistical evidence and make informed decisions.

Hypothesis Example (Using ChatGPT)

Does a New Drug Reduce Blood Pressure?

In this example, we'll demonstrate hypothesis testing using a hypothetical scenario where a pharmaceutical company has developed a new drug intended to reduce blood pressure in patients with hypertension. We'll design and conduct a hypothesis test to determine whether the new drug is effective in reducing blood pressure compared to a placebo.

Hypotheses Formulation
- Null Hypothesis (H0): The new drug has no effect on reducing blood pressure, and any observed differences are due to random chance.
- Alternative Hypothesis (Ha): The new drug effectively reduces blood pressure in patients, leading to a significant difference compared to the placebo.
Data Collection
- We collect blood pressure measurements from two groups of participants: one receiving the new drug (treatment group) and the other receiving a placebo (control group).
- Each participant's blood pressure is measured before and after the treatment period.
Experimental Design
- Randomly assign participants to either the treatment group (receiving the new drug) or the control group (receiving a placebo).
- Ensure blinding to minimize bias, where neither the participants nor the researchers know who is receiving the drug or the placebo.
Data Analysis
- Calculate the mean blood pressure reduction for each group.
- Conduct a hypothesis test, such as a two-sample t-test, to compare the mean blood pressure reductions between the treatment and control groups.
Interpretation of Results
- If the p-value is less than the chosen significance level (e.g., α = 0.05), we reject the null hypothesis in favor of the alternative hypothesis.
- A statistically significant result indicates that the new drug has a significant effect on reducing blood pressure compared to the placebo.
- Conversely, if the p-value is greater than the significance level, we fail to reject the null hypothesis, suggesting no significant difference between the groups.
Conclusion and Recommendations
- If the hypothesis test results support the alternative hypothesis, we conclude that the new drug is effective in reducing blood pressure.
- The pharmaceutical company can proceed with further clinical trials and regulatory approvals for the new drug.
- If the results do not support the alternative hypothesis, the company may need to reevaluate the drug's efficacy or explore alternative treatment options.

import numpy as np
from scipy.stats import ttest_ind

# Simulated blood pressure data
treatment_group = np.array([140, 135, 150, 138, 132])  # Blood pressure reduction after new drug
control_group = np.array([145, 142, 148, 146, 140])     # Blood pressure reduction after placebo

# Perform two-sample t-test
t_statistic, p_value = ttest_ind(treatment_group, control_group)

# Interpretation of results
alpha = 0.05
if p_value < alpha:
    print("The new drug significantly reduces blood pressure compared to the placebo.")
else:
    print("There is no significant difference in blood pressure reduction between the groups.")

# Additional analyses, visualization, and interpretation can be conducted as needed.

We simulate blood pressure reduction data for participants in the treatment and control groups.
We use a two-sample t-test to compare the mean blood pressure reductions between the two groups.
If the p-value is less than the significance level (e.g., 0.05), we conclude that the new drug is effective in reducing blood pressure.
Otherwise, we fail to reject the null hypothesis, indicating no significant difference between the groups.

Errors

In hypothesis testing, there are two types of errors that can occur when making a decision about the null hypothesis: Type I error and Type II error.

Type 1 Error

Type-I (False Positive) error occurs when the sample results, lead to the rejection of the null hypothesis when it is in fact true.

In other words, it's the mistake of finding a significant effect or relationship when there is none. The probability of committing a Type I error is denoted by α (alpha), which is also known as the significance level. By choosing a significance level, researchers can control the risk of making a Type I error.

Type 2 Error

Type-II (False Negative) error occurs when based on the sample results, the null hypothesis is not rejected when it is in fact false.

This means that the researcher fails to detect a significant effect or relationship when one actually exists. The probability of committing a Type II error is denoted by β (beta). Trade-off between Type 1 and Type 2 errors

Where Can Be Hypothesis Testing Applied?

Testing the effectiveness of interventions or treatments: Hypothesis testing can be used to determine whether a new drug, therapy, or educational intervention has a significant effect compared to a control group or an existing treatment.
Comparing means or proportions: Hypothesis testing can be used to compare means or proportions between two or more groups to determine if there's a significant difference. This can be applied to compare average customer satisfaction scores, conversion rates, or employee performance across different groups.
Analysing relationships between variables: Hypothesis testing can be used to evaluate the association between variables, such as the correlation between age and income or the relationship between advertising spend and sales.
Evaluating the goodness of fit: Hypothesis testing can help assess if a particular theoretical distribution (e.g., normal, binomial, or Poisson) is a good fit for the observed data.
Testing the independence of categorical variables: Hypothesis testing can be used to determine if two categorical variables are independent or if there's a significant association between them. For example, it can be used to test if there's a relationship between the type of product and the likelihood of it being returned by a customer.
A/B testing: In marketing, product development, and website design, hypothesis testing is often used to compare the performance of two different versions (A and B) to determine which one is more effective in terms of conversion rates, user engagement, or other metrics.

Hypothesis Testing in ML Applications

Feature selection: Hypothesis testing can help identify which features are significantly related to the target variable or contribute meaningfully to the model's performance. For example, you can use a t-test, chi-square test, or ANOVA to test the relationship between individual features and the target variable. Features with significant relationships can be selected for building the model, while non-significant features may be excluded.
Hyperparameter tuning: Hypothesis testing can be used to evaluate the performance of a model trained with different hyperparameter settings. By comparing the performance of models with different hyperparameters, you can determine if one set of hyperparameters leads to significantly better performance.
Assessing model assumptions: In some cases, machine learning models rely on certain statistical assumptions, such as linearity or normality of residuals in linear regression. Hypothesis testing can help assess whether these assumptions are met, allowing you to determine if the model is appropriate for the data.
Model comparison: Hypothesis testing can be used to compare the performance of different machine learning models or algorithms on a given dataset. For example, you can use a paired t-test to compare the accuracy or error rate of two models on multiple cross- validation folds to determine if one model performs significantly better than the other.

[!WARNING] Questions

What is P-value?

What are different types of Hypothesis Tests.

Differences between Z-test and T-test?

Resources

Learn Web Scraping

Anshul Raj Verma — Sat, 23 Mar 2024 00:00:00 GMT

Web scraping is a very essential tool for programmers to learn to gather data from websites. Specifically, for Data Scientists web scraping is goto tool to gather data from websites. We can use bs4.BeautifulSoup or selenium in Python to scrape any website.

You can see some of my projects where I scraped websites like 99acres.com, flipkart.com, housing.com and gather useful data for my Data Science projects like arv-anshul/campusx-real-estate.

I have learned Web Scraping from YouTube only.

YouTube Videos

Corey Schafer

YouTube Playlists

If you follow/learn these resources then you will understand how do Web Scraping works and how to do it.

My Python Package for Web Scraping

I have done lots of project on Web Scraping but while doing those web scraping projects I doesn't found a good python package to handle/parse cURL command. But I found a package called @spulec/uncurl on GitHub but it is managed badly so that I cloned that project and refactor it well and republished as curler on PyPI.

My Projects Related to Web Scraping

Learning Diary Migration

Anshul Raj Verma — Fri, 22 Mar 2024 12:53:00 GMT

I used to maintain my diary with Obsidian app. And recently I have learned mkdocs-material which is used to create static website using markdown files only.

Obsidian is made to use it as a notes/documents management tool/app but I am just writing my thoughts and journals, nothing else. And for this I can also use VSCode or any other editor to write notes in markdown format. That's when I know about mkdocs-material and I fall for it :heart:.

When I decided to migrate my notes I have lots of notes regarding My Projects, Data Science, FastAPI and more. So I have to decide whether to remove them or keep them, so I decided to merge the similar one and publish them as blogs on the website.

Merged Notes as Blogs

I have published most of the notes as blog except My Projects notes which contains the rough works which I used to do while creating projects. So I completely :fire: removed them and decided to maintain them on the website as projects notes.

If you want to see how the diary looks while I am maintaining it with obsidian. You can also see My Projects Notes there.

diary/v1.0.0

Tree VS Regression Models

Anshul Raj Verma — Thu, 21 Mar 2024 00:00:00 GMT

From CampusX Trail Session

Tree based models and Regression models are widely used Machine Learning models. So more you know about them is better for you. Also, many concepts from these models are borrowed by advance Machine Learning models like Gradient Boosting, XGBoost, etc.

These models are also great choice for interviewers so from these models they ask many interview questions. This blog mainly focuses on tree based models.

Aspect	Decision Trees	Regression Models
Type	Can be used for classification and regression	Used for predictive modeling, predicting continuous values
Decision Boundaries	Bisect the space into smaller regions, fitting lines to divide the space exactly	Focuses on predicting outcomes based on previous data or trends
Interpretability	Easy to understand and interpret due to their flowchart-like structure	Interpretability varies based on the complexity of the model
Advantages	Easy to interpret, visualize, and require minimal data preparation	Effective in predicting continuous values based on historical data
Disadvantages	Prone to overfitting noisy data, especially with deeper trees	May struggle with capturing complex relationships in the data
How They Work	Split the dataset based on data homogeneity, using measures like entropy for classification trees and Sum of Squared Errors for regression trees	Focuses on fitting a curve or surface to the data points to predict continuous values

1. What Are the Main Differences Between Tree Based Models and Regression Models?

Aspect	Tree Based Models	Linear Models
Approach	Tree based models uses Divide & Conquer approach to learn the data essence by making cut into data space to create small spaces which segregate the homogenous data.	Regression models tries to create/fit a line in-between the data space, to get the essence of the data and tries to keep the value of loss function minimum as possible.
Non-Linearity	Tree models can handle both linear and non-linear data.	Regression models performs better with linearly separable data but it fails to capture the essence of non-linear data.
Computational Complexity	Tree models are more complex than linear models, especially training complexity of Ensemble techniques as it finds optimal solution iteratively.	Regression models has an upper edge here because it requires find the optimal co-efficient which minimizes the loss function and for that it uses Gradient Descent techniques.
Effects of Outliers	Tree models are generally robust to outliers because they creates multiple decision boundaries which separates the outliers easily and model doesn't affect much from it.	Regression models are highly sensitive to outliers because when the data has outliers the best fit line will try to fit the outliers values to reduce the loss function. To reduce this sensitivity we have Regularization concepts.
Handling Null Values	Some tree based algorithms/models can handle null values like XGBoost.	Linear model can't handle null values at all. Preprocess the data before training linear models.
Interpretable	Only Decision Trees are interpretable but libraries like SHAP can explain ensemble tree models. RandomForest can be used to calc features importance.	Linear models are very interpretable because it calculates the features co-efficient with target feature.

2. Can You Describe a Real-World Application Where You Would Prefer to Use a RandomForest over a Logistic Regression Model?

We can preffer RandomForest over Logistic Regression in the many scenarios like:

When data has outliers.
When data is hight dimensional.
When data is non-lineear.
When data is imbalanced.
RandomForest algorithm is robust to overfitting while training.
Also, RandomForest can handle categorical features better than logistic regression because you can use OrdinalEncoder to encode categorical feature instead of OneHotEncoder.
By the way, you can use libraries like h2o which can handle null values too.

3. What Is the Impact of Outliers on Decision Tree?

Decision Trees can handle Outliers easily because it segregate them using decision boundaries in the initial steps. However, if the Decision Tree becomes overfitted to a training dataset, it can become more sensitive to outliers, potentially affecting the model's performance. Also, you can Regularization methods to tackle outliers.

Mainly outliers can affect Decision Trees while working with regression problems (only in those leaf-nodes where outliers are present/classified/calculated). Prediction of model is affected in those leaf-nodes where outliers are calculated (this is proven) but this is not the case in classification problems.

4. What Is the Role of Pruning in Decision Tree, What Is post-Pruning and pre-Pruning?

Role of Pruning in Decision Trees

Pruning in Decision Trees is crucial to prevent overfitting and enhance the model's ability to generalize by simplifying the tree structure.
It involves removing parts of the tree that do not contribute significantly to predictive power, making the model more interpretable and effective.

Pre-Pruning

Pre-pruning involves stopping the tree's growth before it fits the entire training set.
It focuses on setting hyperparameters to control the tree's size during construction, preventing overfitting by limiting its complexity.

Post-Pruning

Post-pruning allows the tree to grow fully and then removes nodes that do not add substantial predictive power.
Techniques like cost-complexity pruning are commonly used in post-pruning to simplify the tree by selecting the subtree with the smallest cost based on a complexity parameter and the number of leaf nodes.

[!NOTE]

Pre-pruning: Penalize (by cutting the nodes) the model while training.

Post-pruning: First fully train the model then after penalize the model by cutting down the nodes.

Cost Complexity Function in Decision Tree

The cost complexity function in decision trees is a crucial concept related to pruning techniques. It involves a tradeoff between error (cost) and tree size (complexity) to find an optimal tree. The cost complexity of a tree, denoted as #!math R_{c_p}(T), is the sum of its risk (error) and a "cost complexity" factor #!math c_p multiplied by the tree size #!math T. This function is used in cost complexity pruning to minimize the cross-validated prediction error and prevent overfitting. By adjusting the cost complexity parameter #!math c_p, decision trees can be pruned effectively to improve generalization to test data.

plot_cost_complexity_pruning

5. How Missing Values Are Handled in Tree Based Algorithms like XGBoost?

XGBoost only handle missing values present in Input features. It doesn't handle null values present in Output/Target feature. You have to preprocess or remove the null values of Output feature.

Not completed!

6. What Is the Difference Between ID3, C4.5, and CART Algorithms?

Algorithm	ID3	C4.5	CART
Type	Iterative Dichotomiser 3	Iterative algorithm, extension of ID3	Classification and Regression Trees
Handling of Numeric Attributes	Less effective for numeric attributes	Handles numeric and categorical attributes	Handles numeric and categorical attributes
Splitting Criteria	Information Gain	Gain Ratio	Gini diversity index for classification tests
Pruning	Does not handle pruning	Prunes trees to avoid overfitting	Prunes trees using a complex model with parameters estimated by cross-validation
Binary Tests	Always binary tests	Allows two or more outcomes	Binary tests

7. How Would You Approach a Situation Where Your Tree-Based Model Is Overfitting?

I can apply pruning techniques which penalize the model if it tries to overfit while training. We have many hyperparameters in DecisionTree, RandomForest classes apply pre-pruning on models.

max_depth: The maximum depth of the tree. Prevent the tree to grow after specified depth.
min_samples_split: The minimum number of samples required to split an internal node.
min_samples_leaf: The minimum number of samples required to be at a leaf node. A split point at any depth will only be considered if it leaves at least min_samples_leaf training samples in each of the left and right branches. This may have the effect of smoothing the model, especially in regression.
min_weight_fraction_leaf: The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided.
max_features: The number of features to consider when looking for the best split.
max_leaf_nodes: Grow a tree with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.
min_impurity_decrease: (Best parameter for pruning) A node will be split if this split induces a decrease of the impurity greater than or equal to this value.
class_weight: Weights associated with classes in the form {class_label: weight}. If None, all classes are supposed to have weight one. For multi-output problems, a list of dicts can be provided in the same order as the columns of y.
ccp_alpha: Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen. By default, no pruning is performed.

sklearn.tree.DecisionTreeClassifier

8. Discuss the Role of Shrinkage (Learning Rate) in Boosting Algorithms. How Does It Contribute to Model Performance and Robustness?

The learning rate, also known as shrinkage, plays a crucial role in boosting algorithms by determining how fast or slow a model updates its weights based on the gradient of the loss function.
A fixed learning rate can lead to challenges such as overshooting the optimal point with a high value or slow convergence with a low value, highlighting the importance of dynamic adjustments during training.
Adaptive learning rate schedules, like decay learning rates, gradually reduce the learning rate as training progresses, helping to avoid overshooting the minimum and fine-tune model parameters more precisely.
Decay learning rates can be implemented using various methods like fixed or exponential decay rates, step or inverse decay functions, and are essential for efficient and effective model training, especially in complex and non-convex problems.
By adjusting the learning rate dynamically, decay learning rates ensure that the model progresses effectively towards the optimal solution, balancing the size of steps taken during training to enhance model optimization and robustness.

This question comes under Boosting algorithms which is advance topic.

9. Discuss the Computational Complexity of Training a Decision Tree and How It Scales with the Size of the Dataset.

The computational complexity of training a Decision Tree is influenced by the size of the dataset and the number of dimensions in the feature space.
Heuristic algorithms are commonly used to compute Decision Trees from training data, aiming to minimize the size of the resulting tree, which impacts the computational complexity of the process.
Understanding the computational complexity of training Decision Trees is essential for optimizing algorithms and improving efficiency in machine learning tasks, especially when dealing with large datasets and high-dimensional feature spaces.

10. How Do Tree-Based Algorithms Handle Imbalanced Datasets, and What Are the Implications for Model Performance and Interpretation?

There is a hyperparameter in DecisionTree class called class_weigth where you can assign weights to each class label or you can provide "balanced" which will automatically assign weights to each class labels.

How Do You Manually Assign Weights?

It depends on your domain knowledge or you can use hit & try method.

How Do `"balanced"` Value Assign Weights?

It assigns #!math \frac{1}{n} weight value to each class labels where #!math n is the count of data points present the following class.

Similar Blogs

CampusX Trail Sessions

Anshul Raj Verma — Wed, 20 Mar 2024 08:35:00 GMT

Nitish Sir wants to recruit teachers to teach along in the course and for that he had to evaluate applicants thats why he conducted trail session in DSMP 2.0 course where the selected applicants comes and take a 2 Hours long session on assigned topic (assigned by Nitish Sir). The applicant take seesion and the students had to fill a Google Forms Feedback Form. Applicants are evaluted on the basis of feedback given by the students.

Session are conducted in March, 2024. and all the trial session are awesome and amazing (except some). I have learned many new concepts from them and I've rank them below on the basis of my experiences.

Best Sessions

Model Explainability (by Balaji Chippada)
Solving a banking problem using ML (by Rohan Azad)
NER using NLTK and Spacy (by Md. Amanatullah)
Intro to Power BI (by Vibhore Aggarwal)
Interview Qs on Tree Based Models (by Ajay Sati)
Introduction to Time Series Forecasting (by Ajinkya Mohite)
Interview Questions on Regression (by DP Sharma)
Intro to FastAPI (by Md. Misbahullah Sheriff)
Project Based Interview Sessions (by Rohan Azad)
ML Interview Questions Sessions (by Rohan Azad)

Mediocre Sessions

Intro to PyTorch (by Rishab Tomar)
Latent Dirichlet Allocation (LDA) (by Aman Bansal)
Prompt Engineering (by Saksham Kumar)
Anamoly Detection (by Chandramouli Das)
eKYC using Computer Vision (by Bibek Rauth)
A/B Testing (by Dixit Patel)
Introduction of Langchain (by Sameer Singh)

Bad Sessions

Interview Questions on Statistics (by Chirantan Lonkar)
ResNET Paper Discussion (by Vinod Tiwari)
Multi-Output Multi-Class Classification (by Prabin Kumar Nayak)

Thanks to Nitish Sir and all the Applicants for these amazing sessions.

Zed Code Editor

Anshul Raj Verma — Wed, 20 Mar 2024 07:49:00 GMT

Zed is not just other code editor community is working on it continuously on it. I have been following Zed development for a long time and I also trying it and also trying to understand its difference from Visual Studio Code (VSCode).

I have to say that Zed is ***king faster than VSCode. Yeh itna tez ki lagta hai ki slow karne ka koi setting hota 🫣. But it's UI is not engaging as VSCode's because maybe because it has not all the essential features like (rigorous) Syntax Highlighting, SCM UI Integration, Extension Support, etc. for now 😞.

I'm waiting for Zed to be ready so that I can use it daily without hesitation just like I use VSCode.

Thanks Zed Contrbutors.

I have also mentioned Zed in my Journal.
Configured Zed for python development Journal.

Learn Docker

Anshul Raj Verma — Sun, 17 Mar 2024 00:00:00 GMT

Docker is a platform designed to simplify the process of creating, deploying, and managing applications using containers. Containers enable developers to package an application with all its dependencies into a standardized unit for seamless deployment across different environments.

:hammer_and_wrench: Components of Docker

Dockerfile
.dockerignore
docker-compose.yaml

`Dockerfile`

A Dockerfile serves as a blueprint for building Docker images, which are the executable packages containing everything needed to run an application - code, runtime, system tools, libraries, and settings. Let's break down the components of a Dockerfile and its significance in the context of Docker:

Context on Dockerfile

Foundation of Image Creation: A Dockerfile specifies a sequence of instructions to assemble an image. It starts with a base image (e.g., Ubuntu, Alpine Linux, Python) and then layers additional configurations and dependencies on top of it.
Clear and Reproducible Build Process: Each instruction in the Dockerfile represents a step in the image-building process. These steps are executed in order, and Docker caches intermediate layers, facilitating faster subsequent builds and ensuring consistency across environments.
Key Components of a Dockerfile:
- Base Image: Specifies the starting point for the image.
- Environment Setup: Includes commands to install packages, set environment variables, copy files into the image, etc.
- Application Configuration: Defines how the application should be configured inside the container.
- Startup Commands: Specifies the command to execute when the container starts.

Comprehensive Description of a Dockerfile

A typical Dockerfile consists of several sections:

FROM: Defines the base image. It's the starting point for the image build and often references an official or custom base image from a registry (e.g., FROM python:3.11-slim).
WORKDIR: Sets the working directory inside the container where subsequent commands will be executed.
COPY/ADD: Copies files or directories from the host machine into the container's filesystem. This includes application code, configuration files, etc.
RUN: Executes commands during the image build process. Typically used for installing dependencies, setting up the environment, and other preparatory tasks.
ENV: Sets environment variables within the container. These can define runtime configurations or paths.
EXPOSE: Informs Docker that the container listens on specific network ports at runtime.
CMD/ENTRYPOINT: Specifies the command that should be run when the container starts. CMD is used to provide default arguments for the ENTRYPOINT command, while ENTRYPOINT sets the primary command.

An Example Dockerfile of a Python Project

# Use an official Python runtime as a parent image
FROM python:3.10

# Set the working directory to /app
WORKDIR /app

# Copy the required files and directory into the container at /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Run main.py when the container launches
CMD ["python", "main.py"]

# Build docker container
docker build -t my_python_container .

# Run docker image
docker run -it my_python_container

Importance and Benefits

Portability: Dockerfiles enable developers to create consistent environments, ensuring that applications run identically across various systems and environments.
Reproducibility: By capturing all dependencies and configurations in the Dockerfile, developers can replicate the same environment for development, testing, and production.
Scalability and Efficiency: Docker's containerization allows for quick scaling and resource efficiency, enabling applications to be deployed and managed easily.

A Dockerfile is the backbone of Docker-based applications, providing a clear, reproducible, and scalable approach to building containerized applications. It defines the entire setup and configuration needed to run an application within a container, promoting consistency and ease of deployment across different environments.

Resources to Learn Docker

PiyushGarg YouTube

Visual Studio Code - Docs

`docker`

1. `docker run`

Description: Starts a new container from an image.
Example:
```
docker run -it --name my_container ubuntu:latest
```
- -it: Starts an interactive terminal within the container.
- --name my_container: Names the container as "my_container".
- ubuntu:latest: Specifies the image to use (latest Ubuntu image).

2. `docker ps`

Description: Lists running containers.
Example:
```
docker ps
```
This command shows the containers' IDs, names, status, ports, and images.

3. `docker images`

Description: Lists available images.
Example:
```
docker images
```
Displays a list of all downloaded Docker images along with their tags and sizes.

4. `docker build`

Description: Builds an image from a Dockerfile.
Example:
```
docker build -t my_image:latest .
```
- -t my_image:latest: Tags the image as "my_image" with the "latest" tag.
- .: Specifies the build context (current directory) containing the Dockerfile.

5. `docker stop`

Description: Stops a running container.
Example:
```
docker stop my_container
```
Stops the container named "my_container".

6. `docker start`

Description: Starts a stopped container.
Example:
```
docker start my_container
```
Starts the container named "my_container" that was stopped.

7. `docker rm`

Description: Removes one or more containers.
Example:
```
docker rm my_container
```
Deletes the container named "my_container".

8. `docker rmi`

Description: Removes one or more images.
Example:
```
docker rmi my_image:latest
```
Removes the image "my_image" with the "latest" tag.

9. `docker exec`

Description: Executes a command within a running container.
Example:
```
docker exec -it my_container bash
```
Executes the Bash shell (bash) in the running container named "my_container" interactively (-it).

10. `docker logs`

Description: Retrieves logs from a container.
Example:
```
docker logs my_container
```
Fetches the logs of the container named "my_container".

These commands form the core of Docker usage for managing containers, images, building, starting/stopping containers, and interacting with containerized applications. They're essential for everyday Docker workflows in development, testing, and deployment scenarios.

`.dockerignore`

.dockerignore is a file used to specify which files and directories to exclude when building a Docker image. It works similarly to .gitignore but for Docker. When building an image, Docker uses this file to determine which files should not be included in the context sent to the Docker daemon, thus reducing the image size and build time.

Example

An .dockerignore file might contain entries like node_modules, *.log, or any other files/directories that are not necessary for the image build process.

# Byte-compiled files
__pycache__
*.pyc
*.pyo
*.pyd

# Virtual environments
venv/
env/
.venv/

# Editor/IDE specific files
.vscode/
.idea/
*.sublime-project
*.sublime-workspace

# Logs and temp files
*.log
logs/
*.tmp

# Miscellaneous
.DS_Store
node_modules/
.cache/

# My custom files for practicing
*.arv
arv.*

Differences between `.gitignore` and `.dockerignore`

Aspect	`.gitignore`	`.dockerignore`
Purpose	Specifies files to ignore in Git	Specifies files to exclude in Docker image builds
Associated tool	Git	Docker
File behavior	Excludes files in Git operations	Excludes files during image build
Impact	Affects version control only	Affects image build process
Use case	Manages repository content	Manages files in Docker context
Ignoring patterns	Glob patterns, file names	Glob patterns, file names

.gitignore and .dockerignore serve different purposes despite their similar naming conventions. While both control what files should be ignored/excluded, .gitignore operates within version control systems, allowing certain files not to be tracked. Conversely, .dockerignore is used during image building to exclude unnecessary files from the Docker context sent to the daemon, optimizing the image build process.

Both files use similar syntax (like glob patterns) to specify what should be ignored, but their impact and contexts in which they're utilized differ significantly.

Doubts

How to write Dockerfile efficiently?
How to use .dockerignore?
How to run two apps with one Dockerfile?
How to integrate Environment Variables in Docker and Python project?
Learn about Python images present on Docker like slim, alpine, bookworm, etc.
How to run multiple apps like FastAPI and Streamlit in one go?
How do I integrate MongoDB image in my Python project?

Learn FastAPI

Anshul Raj Verma — Sun, 17 Mar 2024 00:00:00 GMT

A better framework than Flask. Get production-ready code and API. With automatic interactive documentation. Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as Swagger) and JSON Schema.

Features

Automatic docs: Generate documentation for your API automatically.
- Swagger UI: Interactive exploration, call and test your API directly from the browser.
```
http://localhost:8000/docs
```
- Redoc: Read only documentation. You can also download this doc.
```
http://localhost:8000/redoc
```
Response Validation: Use pydantic BaseModel as TypeHint in Python which automatically validate your responses.
Starlette Features: FastAPI is actually a sub-class of Starlette.With FastAPI you get all of Starlette's features (as FastAPI is just Starlette on steroids):
- Seriously impressive performance. It is one of the fastest Python frameworks available, on par with NodeJS and Go.
- WebSocket support.
- In-process background tasks.
- Startup and shutdown events.
- Test client built on HTTPX.
- CORS, GZip, Static Files, Streaming responses.
- Session and Cookie support.
- 100% test coverage.
- 100% type annotated codebase.
Supports Asynchronous programming.

Important Links To Learn FastAPI

FastAPI Tutorials

Asynchronous Programming Tutorials

Some Advice On FastAPI

Use fastapi.APIRouter to separate out different API paths. See mine @arv-anshul/ecommerce-scrapper-api project for example.
If you don't know, check the "In a hurry?" section about async and await in the docs.
Learn builtin asyncio module in python to do Asynchronous Programming in python. See mine @arv-anshul/yt-watch-history project for example.
Use pydantic with FastAPI for data handling of APIs. See this docs section to know more about Pydantic and FastAPI compatiblity.
You can use the fastapi.testclient.TestClient class to test FastAPI applications without creating an actual HTTP and socket connection, just communicating directly with the FastAPI code. Read more about it in the FastAPI docs for Testing - Tutorial.
There are many other advance concepts in API world and some of them are Middleware, Dependency Injection, CORS, etc. For that see the FastAPI docs.

Extra Links around FastAPI

Handle Outliers - Univariate

Anshul Raj Verma — Sun, 17 Mar 2024 00:00:00 GMT

Handling outlier is a big task for data scientist. To handle the outliers we have many different methods to handle them i.e. IQR, Z-score, Mean-Median Imputation, Winsorization, etc. We are going to discuss only univariate methods to handle outliers.

:calendar: I have written this page as notes very time ago; so if there is any mistake please let me know I'll fix it. Thanks 🤗

Deletion Based Approach

IQR

In this method by using Inter Quartile Range(IQR), we detect outliers. IQR tells us the variation in the data set. Any value, which is beyond the range of #!math -1.5 \ast IQR to #!math 1.5 \ast IQR treated as outliers.

The concept of quartiles and IQR can best be visualized from the boxplot. It has the minimum and maximum point defined as #!math Q1 - 1.5 \ast IQR and #!math Q3 + 1.5 \ast IQR respectively. Any point outside this range is outlier.

[!CAUTION] Cons

It delete your many data point because even if there is only one data point in a row is act as outlier for their respective column then the row is being removed which means to remove one outlier you removed many essential data point from your dataset.

import pandas as pd

def apply_iqr_deletion(df: pd.DataFrame, columns: list[str], *tiles: tuple[float, float]):
    """
    Deletes outliers from given columns which are out of range of minimum and maximum percentile values.

    Parameters
    ----------
    df: pd.DataFrame
        DataFrame to be cleaned.
    columns: list[str]
        List of columns to be cleaned.
    *tiles: tuple[float, float]
        Tuples of (minimum percentile, maximum percentile) for each column.

    Returns
    -------
    DataFrame with deleted outliers data points.

    Raises
    ------
    ValueError: If `len(columns) != len(tiles)`.
    """
	if len(columns) != len(tiles):
		raise ValueError('len(columns) != len(tiles)')

	for col, tile in zip(columns, tiles):
		mini, maxi = df[col].quantile[tile]
		df = df[(df[col]>mini) & (df[col]<maxi)]
	return df

If you have less number of outliers in your data then apply apply_iqr_deletion function but if you have many outliers than a threshold value then use apply_iqr_capping function to cap the outliers within a range.

Z-Score

This method assumes that the variable has a Gaussian distribution. It represents the number of standard deviations an observation is away from the mean.

In this method we calculate the z-score with #!math Z = \frac{(x_i - \bar{x})}{\sigma} of the feature then set a threshold (generally as ±3) then remove the data point which are #!math \ge 3 and #!math \le -3.

[!TIP] You can also calculate absolute value of every z-score then just one constraint is required as #!math \ge 3.

[!CAUTION] Cons

It deletes the rows which contains outlier which leads to data loss. And generally, losing the data is not good because it creates bias in the model and you doesn't inference well.

import numpy as np
import pandas as pd
from scipy import stats

def apply_zscore_deletion(df: pd.DataFrame, columns: List[str], threshold: float = 3.0) -> pd.DataFrame:
    """
    Deletes outliers from given columns using z-score method.

    Args:
        df: pd.DataFrame
            DataFrame to be cleaned.
        columns: List[str]
            List of columns to be cleaned.
        threshold: float
            Threshold for z-score.

    Returns:
        pd.DataFrame
            DataFrame with deleted outliers data points.
    """

    if not isinstance(threshold, (float, int)):
        raise TypeError("Threshold must be a float or integer value.")

    z_scores = np.abs(stats.zscore(df[columns]))
    df = df[columns][z_scores <= threshold]
    return df

It uses mean and standard deviation of the population data which is generally not available so we need to apply hypothesis testing to ensure that sample mean and sample standard deviation is being used instead of population parameters.

[!WARNING] Doubt

Why do we calculate Z-Score because it requires population standard deviation which is not available for every for every datasets.

We should use T-Score instead.

Capping Based Approach

Winsorization

It is a way to minimise the influence of outliers in your data by either:

Assigning the outlier a lower weight.
Changing the value so that it is close to other values in the set.

[!TIP] Pros

It doesn't delete the rows where outliers lie instead it clip those outliers with your defined percentile values for each column.

[!CAUTION] Cons

If there is many outlier values in the column/feature then after clipping the distribution of column/feature will change.

import pandas as pd

def apply_winsorization(df: pd.DataFrame, columns: list[str], *tiles: tuple[float, float]):
    """
    Caps outliers in given columns to the minimum and maximum percentile values.

    Parameters
    ----------
    df: pd.DataFrame
        DataFrame to be cleaned.
    columns: list[str]
        List of columns to be cleaned.
    *tiles: tuple[float, float]
        Tuples of (minimum percentile, maximum percentile) for each column.

    Returns
    -------
    DataFrame with capped outliers data points.

    Raises
    ------
    ValueError: If `len(columns) != len(tiles)`.
    """
    if len(columns) != len(tiles):
        raise ValueError('len(columns) != len(tiles)')

    for col, tile in zip(columns, tiles):
        mini, maxi = df[col].quantile[tile]
        df[col] = df[col].clip(mini, maxi)
    return df

Use this method because it uses capping technique to handle outliers.

Important Links

Regularization in ML

Anshul Raj Verma — Sun, 17 Mar 2024 00:00:00 GMT

Regularization is used to solve the problem of overfitting caused while training a ML model. In regularization, the model is penalized for overfitting on train data means whenever model tries to predict on training data it add some penalty to the loss function in term of coefficients of the model.

[!WARNING] What is Overfitting?

A ML model is said to be "overfitting" when it performs well on training dataset, but the performance is comparatively poor on the test/unseen dataset.

[!WARNING] What is Underfitting?

An ML model is said to "Underfitting" when it does not performs well on both the train as well as test dataset.

Types of Regularization

There are three main type of regularization in ML:

L1 Regularization (Lasso)
L2 Regularization (Ridge)
L1 + L2 Regularization (ElasticNet)

L2 Regularization (Ridge)

Ridge Regression is a technique used in regression analysis to tackle the problem of overfitting, particularly when dealing with multiple correlated predictors or features. The L2 regularization technique, also known as Ridge Regression, adds a penalty term to the standard linear regression equation, which helps to mitigate the effects of multi-collinearity.

The objective function for Ridge Regression is given by:

\text{minimize}\left( \sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^{p} x_{ij}\beta_j \right)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \right)

Where:

#!math (n) represents the number of data points.
#!math (p) represents the number of predictors or features.
#!math (x_{ij}) denotes the value of the #!math (j^{th}) feature for the #!math (i^{th}) data point.
#!math (y_i) represents the observed output for the #!math (i^{th}) data point.
#!math (\beta_0, \beta_1, \beta_2, \dots, \beta_p) are the regression coefficients.
#!math (\lambda) is the hyperparameter that controls the regularization strength.

The first part of the equation is the standard least squares regression term, which aims to minimize the squared difference between the predicted and actual output. The second part is the penalty term, which is the sum of squares of the coefficients #!math (\beta) multiplied by the regularization parameter #!math (\lambda).

The hyperparameter #!math (\lambda) controls the trade-off between fitting the model to the training data and preventing overfitting by keeping the coefficients small. A larger #!math (\lambda) leads to a stronger penalty, effectively shrinking the coefficients toward zero. This helps to reduce the model's complexity, making it less sensitive to the noise in the data.

Ridge Regression is a powerful tool in situations where multi-collinearity among predictors exists. By adding this penalty term, it stabilizes the coefficients and reduces their variance, thus improving the model's generalization and robustness when dealing with new, unseen data.

Implementing Ridge Regression involves finding the optimal values for the coefficients by minimizing the combined error and penalty term. Various optimization algorithms such as gradient descent or closed-form solutions can be employed for this purpose.

In conclusion, Ridge Regression with L2 regularization offers a balance between fitting the data and preventing overfitting, making it a valuable technique in the realm of regression analysis.

L1 Regularization (Lasso)

Lasso Regression, short for "Least Absolute Shrinkage and Selection Operator," is a method used in regression analysis to handle overfitting and perform feature selection by adding a penalty term to the standard linear regression equation. The L1 regularization technique in Lasso Regression introduces sparsity by imposing a penalty based on the absolute values of the regression coefficients.

The objective function for Lasso Regression is given by:

\text{minimize}\left( \sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^{p} x_{ij}\beta_j \right)^2 + \lambda \sum_{j=1}^{p} |\beta_j| \right)

Where:

#!math (n) represents the number of data points.
#!math (p) represents the number of predictors or features.
#!math (x_{ij}) denotes the value of the #!math (j^{th}) feature for the #!math (i^{th}) data point.
#!math (y_i) represents the observed output for the #!math (i^{th}) data point.
#!math (\beta_0, \beta_1, \beta_2, \dots, \beta_p) are the regression coefficients.
#!math (\lambda) is the hyperparameter that controls the regularization strength.

The first part of the equation is the standard least squares regression term that minimizes the squared difference between the predicted and actual output. The second part is the penalty term, which is the sum of the absolute values of the coefficients #!math (\beta) multiplied by the regularization parameter #!math (\lambda).

The hyperparameter #!math (\lambda) controls the trade-off between fitting the model to the training data and keeping the coefficients small. In Lasso Regression, the absolute value of the coefficients' sum is used as the penalty. This has the effect of forcing some coefficients to be exactly zero, effectively performing variable selection by eliminating less influential features. The sparsity induced by L1 regularization makes Lasso Regression particularly useful when dealing with datasets with a large number of features, as it can automatically perform feature selection.

Implementing Lasso Regression involves finding the optimal values for the coefficients by minimizing the combined error and penalty term. Various optimization techniques like coordinate descent or sub-gradient methods can be employed to achieve this.

In conclusion, Lasso Regression with L1 regularization is a valuable tool for not only preventing overfitting but also performing automatic feature selection by shrinking certain coefficients to zero. This makes it a popular choice when dealing with high-dimensional datasets and seeking a more interpretable and sparse model.

ElasticNet

This is the combination of both L1 and L2 regularization.

\text{minimize}\left( \sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^{p} x_{ij}\beta_j \right)^2 + \lambda_1 \sum_{j=1}^{p} |\beta_j| + \lambda_2 \sum_{j=1}^{p} \beta_j^2 \right)

Resources

Regression Interview Questions

Anshul Raj Verma — Tue, 12 Mar 2024 00:00:00 GMT

In an interview, questions around regression model is very prominent. So, let's study some interview questions around it.

Assumptions of Linear Regression?

There are about 5 main assumption while training a Linear Regression Model which are:

1. Linear Relationship Between Input And Output Data

Relationship of every input feature must be linear with output feature.

2. Multi-Collinearity

[!WARNING] What is Multi-Collinearity?

Multicollinearity is a phenomena where two or more independent variables are highly correlated. In other words, one predictor variable can be used to predict the value of another. This creates redundant information, skewing the results in a regression model.

Input data must not correalate with each other, they must be independent of each other. One can use VIF{ title="Variance Inflation Factor" } or correlation matrices to know whether their input data is correlated.

See this section for better explanation.

3. Normally Distributed Residuals

[!WARNING] What are Residuals?

Represent the vertical distance between a data point and the regression line. They are the errors of the model which the model can't able to capture while training.

The distribution of the residuals must be normally distributed. One can analyse this using KDE (Kernel Density Estimator Function) or QQ-Plot.

4. Homoscedacity

Homoscedasticity refers to constant variance in a regression model's residuals. Cons include potential bias and inefficiency. Visualize homoscedasticity using scatter plots — residuals vs. predicted values should show an even spread, indicating consistent variance. In Python, seaborn or matplotlib can create such plots for regression diagnostics.

5. No Auto-Correlation Of Error

Autocorrelation of errors in regression models refers to the correlation between the error terms at different time points or observations. Positive autocorrelation indicates that errors in one period are correlated with errors in previous periods. This violates the assumption of independence, impacting model reliability. Diagnostic plots or statistical tests, like the Durbin-Watson test, can assess autocorrelation in regression residuals.

[!INFO] Resources

What are the main Assumptions of Linear Regression? | Top 5 Assumptions of Linear Regression

Presented all 5 assumptions in Notebook

Why multi-collinearity is a problem?

Multicollinearity occurs when two or more independent variables in a regression model are highly correlated. ^^It is bad^^ because it inflates standard errors, leading to unstable and unreliable coefficients. This makes it difficult to isolate the individual effect of each variable, reducing the model's interpretability and predictive accuracy.

Detection: Use correlation matrices or variance inflation factor (VIF). High correlation coefficients or VIF values (>5) indicate multicollinearity.

Removal: Options include excluding one of the correlated variables, combining them, or using regularization techniques like Ridge/Lasso regression that penalize large coefficients.

[!INFO] Resources

Why Multicollinearity is Bad? What is Multicollinearity? How to detect and remove Multicollinearity

Difference b/w "Person Correlation" and Multi-collinearity.

The main difference between Pearson Correlation and Multicollinearity lies in their applications within regression analysis.

Pearson Correlation is a measure of the linear relationship between two numerical variables, ranging from -1 to 1, where 0 indicates no correlation and values close to 1 or -1 indicate a strong linear relationship

On the other hand, Multicollinearity refers to a situation where independent variables in a regression model are highly correlated with each other, which can lead to issues such as inflated coefficients and weakened statistical measures like p-values

While ^^Pearson Correlation focuses on the relationship between two variables, Multicollinearity deals with the interplay among multiple independent variables in a regression context^^, impacting the model's interpretability and reliability.

What is VIF? Why VIF > 5?

VIF (Variance Inflation Factor) is a measure that quantifies how much the variance of an estimated regression coefficient increases if your predictors are correlated. A VIF greater than 5 or 10 is often considered indicative of multicollinearity.

VIF > 5 suggests that the variance of the coefficient estimate is 5 times higher due to multicollinearity, making the estimate less reliable and harder to interpret.

Use VIF in python as from statsmodels.stats.outliers_influence import variance_inflation_factor.

What is R-squared (R²/R2) score?

R-Squared (R²) is a statistical measure used to determine the proportion of variance in a dependent variable that can be predicted or explained by an independent variable.

In other words, R-Squared shows how well a regression model (independent variable) predicts the outcome of observed data (dependent variable).

R-Squared is also commonly known as the coefficient of determination. It is a goodness of fit model for linear regression analysis. Copied From

What Does an R Squared Value Mean?

A R-Squared value shows how well the model predicts the outcome of the dependent variable. R-Squared values range from 0 to 1.

An R-Squared value of 0 means that the model explains or predicts 0% of the relationship between the dependent and independent variables. And a value of 1 indicates that the model predicts 100% of the relationship, and a value of 0.5 indicates that the model predicts 50%, and so on.

Formula for R Squared

\text{R}^2 = 1 - \frac{\text{RSS}}{\text{TSS}}

Symbol	Description
`#!math \text{R}^2`	Coefficient of determination
`#!math \text{RSS}`	Sum of squares of residuals
`#!math \text{TSS}`	Total sum of squares

[!INFO] Resources

What is R Squared? R2 Value Meaning and Definition

How is Adjusted R2 score is different from R2 score?

Adjusted R2 score is different from R2 score in the way they handle the addition of new predictors to a multiple regression model.

While R2 score increases or remains the same as new predictors are added to the model, even if the newly added predictors are independent of the target variable and don't add any value to the predicting power of the model.

On the other hand, Adjusted R2 score only increases if the newly added predictor improves the model's predicting power. It helps determine the goodness of fit for a multiple regression model by considering the number of predictors and the sample size. Adjusted R2 penalizes the model for useless variables, while R2 does not, making Adjusted R2 a more reliable measure of goodness of fit for multiple regression problems.

[!INFO] Resources

Demystifying R-Squared and Adjusted R-Squared

What-if there is one feature related to another then, what should we do with them?

Keep any one of them after checking the correlation with target feature #!math (y) whichever has low correlation; drop them and keep only one.
Use sklearn.decomposition.PCA (with sklearn.compose.ColumnTransformer) to combine them as one and use that transformed column.

What is Regularization? - Why to use Regularization? - What happen in Regularization?

You can use regularization to penalize the model while trining when mode is trying to overfit with training data. Generally, there are three main types of regularization is used L1 (Lasso), L2 (Ridge) and (L1 + L2) ElasticNet.

My blog on Regularization

March Journal

Anshul Raj Verma — Fri, 01 Mar 2024 00:00:00 GMT

Week 10 Journal

❌ No extra learnings. See 2024 February

Week 11 Journal

Learning

Zed: Tried Zed code editor and I feel promising to adapt.
arv-anshul/dotfiles: Updated dotfiles, inspired by corey/dotfiles.
brew: Learned some new things in brew cli.
- brew autoremove: Uninstall formulae that were only installed as a dependency of another formula and are now no longer needed.
- brew deps: Show dependencies for formula. When given multiple formula arguments, show the intersection of dependencies for each formula. By default, deps shows all required and recommended dependencies.
family-tree: New project to "Create family tree using python and export it as JSON or mermaid format". Written tests too.
md-badges: CANCELLED A CLI tool to query shields.io badges simpleicons in a batch.
- I find it hard and irritating to query each badge on badges.pages.dev website. I want to query/select all required badges at once.
Schema Writting for JSON:
- Learned how to write schema for json, yaml and toml files.
- Learning to define schema for the personal website's docs/data/render_yaml folder's YAML files.
- Watched this video by NeuralNine: JSON Schema Validation in Python: Bring Structure Into JSON
- Check the schemas: projects_index-schema.json and other_projects-schema.json
arv-anshul/campusx:
- Wrote python scripts to download course's resources from internet and store them in GitHub repo itself.
- Also added this project on website.

Thoughts

Zed code editor is just great and I want to use it full time but it lacks many features which are very heavy to fulfil. I want features like 🥇 Extensions, 🥈 Jupyter Notebook, 🥉 Git SCM UI and more.
stats: Amazing app for MacOS which let you display your CPU, RAM, Network Usage, etc in menu bar. I knew about it from a YouTube video. I have installed an app from App Store to display my Network Usage in menu bar after a lot of searching on web.
arv-anshul/diary: Cleaning of the diary.
- Remove unnecessary files and notes.
- Combine the important one and publish them as blog on arv-anshul.github.io like FastAPI and Data Science notes.
- Write & Update conventions of every notes and create templates for them.

Week 12 Journal

Learning

arv-anshul.github.io
- Update some componet's layouts of website i.e. Grid Cards, Code Blocks, Buttons.
- Also, figured-out how to integrate devicons as like emoji.
arv-anshul/diary
- Migrate diary notes as blog on website.
- Trying to drop Obsidian for my diary entry and want to use mkdocs (with basic functionalities) for dairy entry.
- Successfully migrated arv-anshul/diary from Obsidian to mkdocs. Compare releases v1.0.0...v1.1.0.
- Wrote custom snippets for Dev Journal(s) and Thoughts entries in VSCode.
arv-anshul/campusx
- Modified scripts to infer filename from response's headers. Filename is available in "content-disposition" headers key.
- See new "Release v0.8.0: Infer resources filename" on GitHub.
blog
- Tree VS Regression Models

Thoughts

:question: Should I drop Obsidian for my diary entry?
- (NO) because Obsidian comes with many other functionalities like Canvas, Extensions, Templates.
- (NO) because Obsidian is made for managing notes and documents (which is a very big plus+ point).
- (YES) because mkdocs will work as a website which you can make public (if you wish for).
- (YES) because you doesn't have to care about a separate app's constraints (obsidian.app).
Setting up tags - Material for MkDocs
- Learn how to setup tags (as pro).
- Also, figure out how does squidfunk render those "versions", "default values", "sponsors only" cool badges.
- Also see this web page
  - There is an example for tags plugin.
  - Learn how did this works?
raycast
- Amazing app for developers. Just use keyboard to perform all important tasks.
- Best Features: Floating Notes, Snippets, Keyboard Shortcuts, :tada: Confetti, Window Management, Shut Down
- Best Extensions: Arc, Shell Command, Google Translate, Perplexity
- I have created some snippets which helps me to write them very easily like:
  - !gha: https://github.com/arv-anshul
  - !web: https://arv-anshul.github.io
  - !linkedin:
```
🔗 Github Profile: https://github.com/arv-anshul
🔗 Personal Projects: https://arv-anshul.github.io/projects
🔗 Website: https://arv-anshul.github.io
🔗 Blogs: https://arv-anshul.github.io/blog

🙏 Thank You!
```
- :disappointed: But it takes too much memory approx. ~300MB in RAM.

Week 13 Journal

Learning

pw-api
- The project doesn't works now due to some major changes in pwskills.com website's APIs.
- Added a warning regarding this in README.md
notebooks/spotify-analysis/pl-tracks-using-api.ipynb
- I want to analyse the Golden Era 🎵 playlist by @Pushpam on Spotify. To reduce the tracks count because it consists ~300 tracks.
- I've used Spotify Web API to fetch data of the playlist.

Co-author a Git Commit

You can co-author a git commit by specifying the author name and email in commit body.

✨ Learn commit co-authoring  # Commit message

Co-authored by: arv-anshul <arv.anshul.1864@gmail.com>  # Specify who co-authored this

blog/hypothesis-testing-introduction
- Published new blog on Hypothesis Testing.

Thoughts

DeepLearning.ai
- This website has some free short course around LLMs and GenAI.

Spotify Analysis

Anshul Raj Verma — Thu, 22 Feb 2024 00:00:00 GMT

[!NOTE] How to get your Spotify data?

You can request your Spotify data from official website.

Know more about your personal data on Spotify.

<details> <summary>Spotify data file structure</summary>

spotify-data
├── Read_Me_First.pdf         # Introductory document
├── Follow.json               # Data about user's followers
├── Identifiers.json          # Identification information (🙅 Do Not Share!)
├── Identity.json             # User identity details (🙅 Do Not Share!)
├── Inferences.json           # Inferred data from user activity
├── Marquee.json              # Marquee-related information
├── Payments.json             # Payment details (🙅 Do Not Share!)
├── Playlist1.json            # Details about a specific playlist
├── SearchQueries.json        # User's search queries
├── StreamingHistory0.json    # User's Streaming history part 1
├── StreamingHistory1.json    # User's Streaming history part 2
├── Userdata.json             # General user data (🙅 Do Not Share!)
└── YourLibrary.json          # User's Spotify library details like likes, albums, artists, etc.

</details>

Philosophy

I want to analyze my Spotify's Streaming History in a way from which I can know my listening pattern over the time. The way Spotify give the Spotify Wrapped at the end of every year.

The Dashboard

[!CAUTION] Currently not developed!

If you want to join for this contact me on my socials LinkedIn E-Mail

There has been a dashboard (using Streamlit) where other users can upload their StreamingHistory.json files to see analysis on their history.

Some Awesome Insights

To gather insights from json files I've used polars and their builtin .plot accessor which uses hvplot library under-the-hood.

See the Jupyter Notebooks to see all the insights.

<details> <summary>Reference</summary>

Short Code	Description
T/A	Track/Artist
T/As	Track/Artist(s)
P/A	Playlist/Album
P/As	Playlist/Album(s)

These shortcodes used below for better readability.

</details>

Using `StreamingHistory.json`

Contains User's Streaming History with trackName (Streaming Track Name), artistName (Streaming Artist Name), msPlayed (Milliseconds Played) and endTime (When the track ends (as datetime)).

[x] Top T/As in whole dataset.
[x] Top T/As in each month.
[x] Monthly most listened Tracks and Artists.
[x] First day when T/A was played.
[x] No. of distinct T/As listened in each month/year.
[x] A T/A streaming in bar plot (which shows how you stream that during time-to-time).
[x] Which daytime user listen most and whom.
[x] Tracks which have listened most times in a day.
[x] Tracks streaming streak (by day/week).
[x] T/As which only played once
[x] Dates when user does not played any track

Using `Playlist.json`

Contains User's Created Playlist Data with all the tracks added in the playlist.

[x] No. of T/As and Albums in each Playlist
[x] Playlist MinutesPlayed
[x] Most streamed P/As
[x] Check any Track present in multiple Playlists
[x] Streaming timeline of P/As (with plot.line())
[x] Playlist-wise top T/As

Using `YourLibrary.json`

[!WARNING] Currently Working!

Using `Marquee.json`

[!WARNING] Currently Working!

Starship Prompt

Anshul Raj Verma — Wed, 21 Feb 2024 00:00:00 GMT

<figcaption> I have introduced to Starship Prompt a week ago and it makes my shell prompt amazing. </figcaption>

[!TIP] Prerequisites

Install Starship Prompt from official documentation.

Setup your shell (bash, zsh or fish) to use Starship from official documentation

[!NOTE]

I am using MacOS. So, for other OS the paths may differ.

I have used Nerd Fonts thats why there are some symbol which may not appear as they are.

I have defined some conventions to define my starship.toml file. If you want to know them read the sections where I describe my conventions.

:wrench: Customization

You can customize your prompt for each programming languages you use like Python, JavaScript, Rust, and more. See official documentation to know more.

Example: Python

[python]
symbol = ""
style = "arv_python"
format = "[](fg:$style)[$symbol( $version)[( \\($virtualenv\\))](bold bg:$style)](bg:$style)[](fg:$style)"

Example: Docker

[docker_context]
symbol = ""
style = "arv_docker"
format = "[](fg:$style)[$symbol ($context)](bg:$style)[](fg:$style)"

Conventions

Enclosed Modules: Each modules enclose with circular end.

:art: Color Palette

Did you see style = "arv_python" and style = "arv_docker" in above examples. Those are my custom defined palette as "arv-anshul".

palette = "arv-anshul"

[palettes.arv-anshul]
arv_dir = "203"
arv_docker = "026"
arv_git = "063"
arv_python = "028"
arv_custom = "236"

<details> <summary>Print ANSI Colormap</summary>

# Use this function to print ANSI colormap
function colormap() {
    range_start=${1:-1}
    range_end=${2:-255}

    for i in $(seq $range_start $range_end); do
        echo -en "\e[48;5;${i}m  ${(l:3::0:)i}  \e[0m "
        [[ $((i % 10)) -eq 0 ]] && echo
    done
    return 0
}

</details>

If you doc't want to use ANSI color format then you also use starship's pre-defined colors: black, red, green, blue, yellow, purple, cyan, white. You can optionally prefix these with bright- to get the bright version (e.g. bright-white). See in documentation

palette = "arv-anshul-color"

[palettes.arv-anshul-color]
arv_dir = "bright-red"
arv_docker = "bright-blue"
arv_git = "blue"
arv_python = "green"
arv_custom = "black"

Conventions

Palette Preffix: Palette's keys must have a preffix (in my case it is "arv_").
ANSI Codes: I have defined colors in ANSI Codes. See this Gist to know more about ANSI Codes.
Own Color Palette: I have defined programming language-wise (or module-wise) colors which makes easy to change/manipulate the color of any language.

You can also create your own custom color palette in ~/.config/starship.toml.

:people_hugging: Extra customization with custom modules

[custom.github]
detect_folders = [".github"]
format = "[$symbol]($style)"
style = "bg:arv_custom"
symbol = " "

[custom.mkdocs]
detect_files = ["mkdocs.yaml", "mkdocs.yml"]
detect_folders = ["docs"]
format = "[$symbol]($style)"
style = "bg:arv_custom"
symbol = "󱔗 "

[custom.vscode]
detect_folders = [".vscode"]
format = "[$symbol]($style)"
style = "bg:arv_custom"
symbol = "󰨞 "

Conventions

I have used `custom` modules to just show some desirable icons in the prompt but you can do a lot of thing using `custom` modules _(the possibilities are endless)_.

Show Custom Icons: I used custom modules to show icons by detecting files and folders. For example, prompt will show icons when .github folder is present in the current directory.

Refer to official documentation to know more about custom modules.

[!TIP] From official documentation

Issue #1252 contains examples of custom modules. You can go there for inspiration and if you have an interesting example not covered there, feel free to share it there!

<figcaption>If you want to make prompt to look like mine! Click below</figcaption>

starship.toml

CampusX Resources

Anshul Raj Verma — Fri, 16 Feb 2024 00:00:00 GMT

<figure> <a href="https://arv-anshul.github.io/campusx"> <img src="https://github.com/arv-anshul/campusx-dsmp/raw/main/docs/data/assets/home.png" title="Website Home Page" class="rounded border border-outline"> </a> <figcaption>Get all the resources like Notes and Notebooks provided in CampusX Courses.</figcaption> </figure>

Praise by Nitish Sir

Links to Resources

Downloaded Resources: Resources are uploaded in my GitHub repo as files. You can get all the course's resources like .pdf, .ipynb, .docx, .pptx, .xlsx and .py files.
See Resources: Resources are listed on a web page where you can access the content descriptions for all sessions, with the teacher providing helpful links to enhance your understanding of each session.

Clustering Algorithms in ML

Anshul Raj Verma — Tue, 06 Feb 2024 00:00:00 GMT

Applications of Clustering

Customer Segmentation: To show personalized ADs to customers.
Data Analysis: Perform analysis to each cluster after performing clustering on the whole dataset.
Semi Supervised Learning: Google Photos uses this technique to identify person's face and put them into a separate folder.
Image Segmentation: You can create segments in photos to represent different objects in the photo.

KMeans

Working Steps

You tell the algorithm how many clusters are there in the data (this is assumption which is taken before initialization).
The cluster's centroids are initialize with some random values.
The distance of each data point is calculated with each cluster and then the data points are assigned to nearest clusters.
After data points assignment to clusters, the centroid of the clusters is being calculated.
If the centroid's position is:
- Same as before then the algorithm stops.
- Not same as before then the STEP 3 is being re-calculated and the process goes on.

graph TD
  subgraph Initialization
    A(Initialize Centroids)
    B[Calculate Distance and \n Assign Data Points]
  end

  subgraph Iterative Process
    C[Update Centroids]
    D[Reassign Data Points]
  end

  subgraph Convergence Check
    E[/Converged?/]
  end

  A -->|Random Initialization| B
  B -->|Assign Nearest Centroid| C
  C -->|Recalculate Centroids| D
  D -->|Assign New Clusters| E
  E -->|Yes| stop{Stop}
  E -->|No| B

Inertia also known as within-cluster sum of squares (WCSS) in the context of K-Means clustering, is a measure that quantifies the compactness of clusters. It is calculated as the sum of the squared distances between each data point in a cluster and the centroid of that cluster.
Elbow Method is way to decide the number of clusters present in a data. However, this is not a very good method to estimate clusters but it there to help you for that.

Assumptions of KMeans

Spherical Cluster Shape: K-means assumes that the clusters are spherical and isotropic, meaning they are uniform in all directions. Consequently, the algorithm works best when the actual clusters in the data are circular (in 2D) or spherical (in higher dimensions).
Similar Cluster Size: The algorithm tends to perform better when all clusters are of approximately the same size. If one cluster is much larger than others, K-means might struggle to correctly assign the points to the appropriate cluster.
Equal Variance of Clusters: K-means assumes that all clusters have similar variance. The algorithm uses the Euclidean distance metric, which can bias the clustering towards clusters with lower variance.
Clusters are Well Separated: The algorithm works best when the clusters are well separated from each other. If clusters are overlapping or intertwined, K-means might not be able to distinguish them effectively.
Number of Clusters (k) is Predefined: K-means requires the number of clusters (k) to be specified in advance. Choosing the right value of k is crucial, but it is not always straightforward and typically requires domain knowledge or additional methods like the Elbow method or Silhouette analysis.
Large n, Small k: K-means is generally more efficient and effective when the dataset is large (large n) and the number of clusters is small (small k).

Resources to Learn From

Introduction

Objective: Divide data into k clusters.
Algorithm: Iterative process that minimizes the sum of squared distances between data points and their assigned cluster centroids.

Algorithm Steps

Initialization: Randomly select k initial centroids.
Assignment: Assign each data point to the nearest centroid.
Update Centroids: Recalculate centroids based on assigned points.
Repeat: Iteratively repeat assignment and centroid update until convergence.

Advantages

Simple and computationally efficient.
Scales well to large datasets.

Disadvantages

Sensitive to initial centroid selection.
Assumes clusters are spherical and equally sized.

DBSCAN

Density-Based Spatial Clustering of Applications with Noise

DBSCAN is a clustering algorithm commonly used in data mining and machine learning. It's particularly effective in identifying clusters of arbitrary shapes and handling noise in the data.

DBSCAN does not require the number of clusters (like KMeans) to be specified beforehand and can discover clusters based on the density of data points.

Important Terms

Core Points: A data point is considered a core point if there are at least "min_samples" data points (including itself) within a specified distance, usually denoted as "epsilon" (ε). Core points are the central points of clusters.
Border Points: A data point is considered a border point if it is within the specified distance (ε) of a core point but doesn't have enough neighboring points to be considered a core point itself. Border points are part of a cluster but are not central to it.
Noise Points: Data points that are neither core points nor border points are considered noise points or outliers. These points do not belong to any cluster.

Working Steps

Initialization: Select an arbitrary data point that has not been visited.
Density Query: Find all data points within distance ε from the selected point.
Core Point Check: If the number of points found is greater than or equal to "min_samples", mark the selected point as a core point, and a cluster is formed.
Expand Cluster: Expand the cluster by recursively repeating the process for all the newly found core points.
Next Point Selection: Choose a new unvisited point and repeat the process until all data points have been visited.

Resources to Learn From

Introduction

Objective: Identify clusters based on dense regions in data space.
Algorithm: Utilizes density information, considering data points as core, border, or noise.

Algorithm Steps

Density Estimation: Define ε (eps) and minimum points for a neighborhood.
Core Points: Identify dense regions with at least 'minPts' neighbors within ε.
Expand Clusters: Connect core points and expand clusters with border points.

Advantages

Doesn't assume spherical clusters.
Can find clusters of arbitrary shapes.
Robust to outliers.

Disadvantages

Sensitivity to parameter settings (ε, minPts).
Struggles with varying density clusters.

Gaussian Mixture Models

Gaussian Mixture Models (GMMs) are probabilistic models used for clustering and density estimation. Unlike k-means, which assigns data points to a single cluster, GMMs allow each data point to belong to multiple clusters with different probabilities. GMMs assume that the data is generated from a mixture of several Gaussian distributions.

Here are the key concepts associated with Gaussian Mixture Models:

Gaussian Distribution (Normal Distribution): A probability distribution that is characterized by its mean (μ) and standard deviation (σ). The probability density function of a Gaussian distribution is given by:

f(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)

Mixture Model: A combination of multiple probability distributions. In the case of GMMs, these are Gaussian distributions.
Parameters of GMM:
- Weights #!math (πi): The probabilities associated with each component (Gaussian distribution). They represent the likelihood of a data point belonging to a specific cluster.
- Means #!math (μi): The mean values of the Gaussian distributions.
- Covariance Matrices #!math (Σi): The covariance matrices representing the shape and orientation of the Gaussian distributions.
Probability Density Function of GMM:

P(x) = \sum_{i=1}^{k} \pi_i \cdot \mathcal{N}(x; \mu_i, \Sigma_i)

where #!math \mathcal{N}(x; \mu_i, \Sigma_i) is the probability density function of the #!math i^{th} Gaussian distribution.

Working Steps

Initialization: Initialize the parameters of the model, including the weights, means, and covariance matrices.
Expectation-Maximization (EM) Algorithm:
- Expectation Step (E-step): Compute the probabilities that each data point belongs to each cluster (responsibility).
- Maximization Step (M-step): Update the model parameters (weights, means, covariance matrices) based on the assigned responsibilities.
Convergence: Repeat the E-step and M-step until the model converges, i.e., the parameters stabilize.
Prediction: Once trained, the model can be used to predict the cluster assignments or estimate the density of new data points.

GMMs are flexible and can model complex data distributions. They are widely used in various applications, such as image segmentation, speech recognition, and anomaly detection.

Resources to Learn From

Introduction

Objective: Model data as a mixture of Gaussian distributions.
Algorithm: Probability-based approach using the Expectation-Maximization (EM) algorithm.

Algorithm Steps

Initialization: Assign initial parameters for Gaussian distributions.
Expectation Step: Estimate probability of each data point belonging to each cluster.
Maximization Step: Update parameters based on the expected assignments.
Repeat: Iteratively repeat the E-M steps until convergence.

Advantages

More flexible in capturing different cluster shapes.
Provides probabilistic cluster assignments.

Disadvantages

Sensitive to initialization.
Computationally more expensive than K-Means.

February Journal

Anshul Raj Verma — Thu, 01 Feb 2024 00:00:00 GMT

Week 5 Journal

Learning

yt-watch-history: After 3 iterations I have understand how to deal with ML models.
yt-watch-history: New PR for channel_reco in frontend. This PR just take subscription data and shows the recommendation of selected channel.

Thoughts

PROJECT IDEA: I am thinking about to create a Python Package from which you can scrape website using cURL.
In yt-watch-history, I don't want to give so much time on frontend instead more focus on backend.

CRITICISM: If you don't focus on frontend then it is bad to present and maintain. That's why you should also take care of frontend part.

Week 6 Journal

Learning

campusx: I have built new project where you can get all the notes and links provided in the description of sessions of DSMP course.
- I have created this in almost 3 days.
- Done many things like follow conventional commits, written tests and more. See here
I have released v0.1.0 of campusx but I use 🤖 ChatGPT to generate release note (using all commits) and I don't think that it is good release note like I see for other projects.
🤓 I want to learn how to write and manage Release Notes and CHANGELOG.

Thoughts

Conventional Commits: For a long time I have been thinking about to follow a convention for GIT commits but where-ever I check I don't understand then I came across vivaxy.vscode-conventional-commits a VSCode extension and now I know how to write better commits. 😉
Mkdocs: mkdocs-material is a very powerful tool to create static web pages using markdowns.
🎉 The Open Source Contributions of Indians ft. @ApnaCollege.
I am thinking about to switch to pixi or rye because I also want to manage my projects like never before; like professional 🤯.

Week 7 Journal

Learning

rye: My new way to manage projects like pro. I have used it and this is very convenient to use and manage.
rye: I have raised issue #639 while configuring the campusx project. For more info see issue itself.
astral-sh/uv: Another blazing fast tool for python devs to manage their packages (written in 🦀 Rust).
- 🤯 This is actually powerful (I have seen it 👀). It is almost 8x faster than pip when I build my yt-watch-history project using docker-compose it takes 🐢 ~200 seconds with pip but now it takes 🐇 ~25 seconds.
starship: Another blazing fast tool for terminal prompts (written in 🦀 Rust).
- 🎉 Yay, I have learned and configured my terminal prompt using starship.
Social Cards: I have added social card preview on the website. Here, I have hard coded the <meta> tags (which is not recommended). You can use social builtin plugin of mkdocs-material.

Thoughts

Praise by Nitish Sir for campusx project. See my LinkedIn Post on this.
yt-watch-history: I have been configuring this project with rye and there are many challenges I have faced while doing this like:
1. This is a virtual project and there are some limitations comes with virtual packages check the issue #639.
2. This project contains two separate repos backend and frontend which make this a monorepo and for now rye does not well support a monorepo because I have to create one workspace level environment for both repos (and I don't kinda like this).

Week 8 Journal

Learning

Starship: Now I have learned starship in a way that I can customize my prompt easily however I want to be. I written a blog a on it to. You can check my starship.toml on Github.
notebooks/spotify-analysis: I have started a new project which analyses User's Spotify's Streaming History data and give insights from them.
- I have done some analysis on StreamingHistory and Playlist dataset and gain some interesting insights from it.

Thoughts

backdropbuild.com: I have received a mail from @JoeyDeBruin to participate in their program (which is a kind of Hack-a-thon) and mentioned my project yt-watch-history because he thinks that the project good fit their program (and I am doing this project to learn MLOps concepts like Docker, MLFlow).

[!NOTE]

Hey Joey, Thanks to appreciate my project yt-watch-history and also for considering me to participate in your program.

Honestly, I am just learning "MLOps" concepts while building this project and for now, I am not sure that I am a good fit for your program. And by keeping this in mind I am not able to join but I am sure that I'll join it in future for sure.

Thanks Joey <br/> Anshul Raj Verma

Week 9 Journal

Learning

😞 Not Much!
But I have watched some sessions of the DSMP 2.0.

Thoughts

Before Wedding: I am going to wedding for approx. 3-4 days.
After Wedding: I have absent for 11 days from 25 Feb — 6 Mar.

Week 10 Journal

Learning

rye transferred to astral-sh Github Organization.
ruff released it v0.3.0 with new formatting rules 2024-02. Read the Blog
I have simplified the complex url of blogs in my website.

Thoughts

I have to bring my laptop to where I go I go because without it I lose many things and make it hard to catch up.

Overview: Decision Tree

Anshul Raj Verma — Mon, 29 Jan 2024 00:00:00 GMT

Decision tree is a very crucial algorithm in ML world because using this algorithm there are many important and some of the best algo of ML is made upon like RandomForest and Xgboost.

That's why we going to take a overview of Decision Tree in this blog.

Anatomy of Decision Trees

A decision tree is a popular machine learning algorithm used for both classification and regression tasks. It is a tree-like model composed of nodes, where each internal node represents a decision based on a feature, each branch represents the outcome of that decision, and each leaf node represents the final prediction.

Root Node: The topmost node that makes the initial decision.
Internal Nodes: Decision nodes that split the data based on a particular feature.
Branches: Outcomes of decisions, leading to the next set of nodes.
Leaf Nodes: Terminal nodes providing the final predictions.

graph TD
    RootNode(Root Node) --> |Decision Feature 1| InternalNode1(Internal Node 1)
    InternalNode1 --> |Decision Feature 2| InternalNode2(Internal Node 2)
    InternalNode1 --> |Decision Feature 3| InternalNode3(Internal Node 3)

    InternalNode2 --> |Decision Feature 4| LeafNode1(Leaf Node 1)
    InternalNode2 --> |Decision Feature 5| LeafNode2(Leaf Node 2)

    InternalNode3 --> |Decision Feature 6| LeafNode3(Leaf Node 3)
    InternalNode3 --> |Decision Feature 7| LeafNode4(Leaf Node 4)

Loss Functions

Decision trees use various loss functions depending on the task:

Classification: Commonly use metrics like Gini impurity or cross-entropy.
Regression: Typically use mean squared error.

Gini Impurity in Decision Trees

How Gini Impurity Works?

Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled. For a binary classification problem, the Gini impurity (Gini index) for a node is calculated as follows:

\text{Gini(D)} = 1 - \sum_{i=1}^{c} (p_i)^2

Where:

#!math (D) is the dataset at the node.
#!math (c) is the number of classes.
#!math (p_i) is the probability of choosing a data point of class #!math i.

Example

Consider a node with 30 samples, distributed among two classes (A and B) as follows:

Class A: 15 samples
Class B: 15 samples

\text{Gini(D)} = 1 - \left(\left(\frac{15}{30}\right)^2 + \left(\frac{15}{30}\right)^2\right)

\text{Gini(D)} = 1 - \left(\frac{1}{4} + \frac{1}{4}\right)

\text{Gini(D)} = 1 - \frac{1}{2} = \frac{1}{2}

The goal during the tree-building process is to minimize the Gini impurity at each node.

Watch StatQuest video on Decision Tree

Gini Impurity Clearly Explained

Advantages

Interpretability: Decision trees provide a transparent and intuitive representation of decision-making. This makes them valuable for communication with non-technical stakeholders. A data scientist can easily explain the logic behind predictions, fostering trust in the model.
No Feature Scaling Required: Decision trees are not affected by the scale of features. This means that data scientists don't have to spend time and effort on feature scaling, making the preprocessing phase simpler and more straightforward.
Handle Mixed Data Types: Decision trees can handle both numerical and categorical data without the need for one-hot encoding. This is advantageous when dealing with datasets that contain a mix of data types.
Require Less Data Preprocessing: Decision trees are less sensitive to outliers and missing values compared to some other algorithms. This can save time during the data cleaning and preprocessing stages.
Feature Importance: Decision trees provide a natural way to assess the importance of different features in the prediction. Data scientists can easily identify which features contribute more significantly to the model's decision-making process.

Disadvantages

Overfitting: Decision trees are prone to overfitting, especially when the tree is deep and captures noise in the training data. Data scientists need to carefully tune hyperparameters, such as the tree depth, to prevent overfitting and ensure better generalization to new data.
Instability: Small variations in the training data can lead to different tree structures. This instability can make decision trees sensitive to the specific training dataset, requiring caution when deploying the model in different environments.
Not Suitable for Complex Relationships: Decision trees may not capture complex relationships in the data as effectively as more advanced algorithms.
Biased Toward Dominant Classes: In classification problems with imbalanced classes, decision trees can be biased toward the dominant class. This can impact the model's performance, especially when accurate predictions for minority classes are crucial.
Not Well-Suited for Regression: While decision trees are excellent for classification tasks, they may not perform as well for regression tasks on continuous data. Other algorithms like linear regression or support vector machines might be more appropriate in such cases.

Conclusion

In conclusion, decision trees are powerful tools with a clear structure and interpretability. Understanding their components, loss functions, and characteristics will help you effectively apply and interpret this versatile algorithm.

ContentType Prediction

Anshul Raj Verma — Sat, 27 Jan 2024 00:00:00 GMT

I am building the ContentType Prediction System from scratch, this it is robuster, flexible and scalable.

<figure> <img class="bg-gray-800 dark:bg-transparent" src="../assets/ml-system-diagram.png" alt="ml-system-diagram.png" /> <figcaption>ML System Diagram for "ContentType Prediction System"</figcaption> </figure>

I have created custom sklearn transformers to transform the datasets. But it is not that good.
I also implemented the model monitoring part using abstraction classes. I do monitoring using mlfow.
I also write scripts for the reference about how to monitor, train and predict models, through this I want to give you some idea that how does this pipeline works. These scripts are for reference only.

Custom Transformers Using `sklearn`

Yesterday, I have learned how to create a custom transformer using sklearn API.

I find it very useful and very elegant way to create pipelines with it. They are very simple to use and implement when you get it right.

[!IMPORTANT] A high level info about custom transformers.

Create a class which inherit two sklearn classes from sklearn.base module TransformerMixin and BaseEstimator.

Now, you have to define fit and transform methods in your class.

And you are ready to use this custom transformer.

Remember this is not a fully pleged custom class because there are numerous things you have to keep in mind while making a custom transformer using sklearn API.

[!IMPORTANT] References

Professional Preprocessing with Pipelines in Python - NeuralNine

Developing a Custom Scikit-learn Transformer and Estimatior - Ploomber

Developing custom scikit-learn transformers and estimators

Monitoring With `mlflow`

1st Draft

I have think a custom monitoring pipeline where you pass the model and params with the training and testing set. Then it calculate the score and log it with mlflow.

<figure> <img src="../assets/mlflow-monitoring-draft-1.png" alt="mlflow-monitoring-draft-1.png" /> <figcaption>MLFlow UI showing scores of multiple trained models on bar chart.</figcaption> </figure>

ML System Doubts

Anshul Raj Verma — Wed, 24 Jan 2024 00:00:00 GMT

I've been working on a project of Machine Learning where I am using Docker to containerise my applications (frontend and backend). But I'm facing difficulties while using ML models in the containers.

[!WARNING] Question

How to train the model and also use MLFlow for model monitoring?

I don't know how to integrate the ML models in the containers.

Should I deploy my models in cloud and from there I can fetch the models for prediction?

Should I add the models into the container from which I can easily make prediction?

MLFlow

In my project Project on GitHub yt-watch-history, I am using MLFlow (but not using it also) means I have written code to train the model with MLFlow but I can also train without it (and I always use this only).

[!WARNING] Question

Should I train the model using MLFLow or just do the model monitoring while Hyper-parameter Tuning?

Containers

What I am doing right now?

I am training the models before starting the container and after, training the models and starting I do predictions using those models by mounting the directories (where models are stored).

What should I have to do?

I can train the model locally and deploy it on the cloud and then fetch and store* the model while prediction.
I can containerise the model too with the codes which makes it easy to use and locate. But comes with many disadvantages like scalability, container's size, model availability/redundancy and more.

Databases

I have been using this as optional step because I never use database model training purpose. I prefer to fetch the data from database and store them into local files and then use them for all the steps.

[!WARNING] How should I use them for model training?

Should I always fetch data from databases for training?

Should I fetch the data once and do all the required steps like data preprocessing, data transformation, features selection, model training and all? I think this is a better approach.

YouTube Watch History

Anshul Raj Verma — Sun, 21 Jan 2024 00:00:00 GMT

Project Introduction: Hello, I am Anshul Raj Verma and I am exited to share my end-to-end Machine Learning project where I've used FastAPI, Streamlit, MongoDB and Docker as tech stack. Also this is version 2 of the project because the version 1 got very complicated and it's hard to modify and refactor the codes there that's why created new version 2 where I am trying to keep better attention on project architecture.
Project Overview: This project consists a streamlit app where you can upload your YouTube Watch History Data to see insights on your viewing pattern. Your data will go through an ML Model which predicts the ContentType of each uploaded video. The app fetches more details of each video through YouTube API. There is also a Channel Recommender System in project which recommend you similar channels on the basis of channel's videos title and tags they had used.
Components of Project: The project is divided into three major components Backend API, ML & Frontend.
- Backend API: This is a FastAPI app which interacts with MongoDB database where YouTube videos details were stored and it also fetches YouTube videos details from official YouTube Data API (for this you require the API_KEY).
- ML: Here the code for ML Model were present through they will get trained and do predictions on user's uploaded data after some preprocessing. The ML Models get served as API through a FastAPI app.
- Frontend: Here all the above components meets and work together to show awesome insights on user's uploaded data. This is a streamlit web app where users can upload their watch history data and see insights. Here above API services were called to fetch videos details from official YouTube API, to store data in database, to make predictions using ML Models, to recommend channels and etc.
Project Architecture: I have created some diagrams to showcase the project's architecture and for that created a dedicated page.
Containerization with Docker: All the three components (Backend API, ML & Frontend) of project were containerized using docker and used docker compose to wrap all three images in a container. uv is used to install packages in docker images. mongodb image is used as database for the project.
More in Project: As I am learning MLOps concepts I am trying to implement them in this project because planning to add DVC and MLFlow into the project.

Channel Recommender System

Anshul Raj Verma — Sun, 21 Jan 2024 00:00:00 GMT

I've built contentType prediction pipeline using videos titles. Now, I am thinking that what if I can recommend similar channels on the basis of their subscribed channels. I can recommend channels using channel's videos titles and videos tags.

Training Pipeline

Data Ingestion and Preprocessing

System import data from two types of sources db (database) and file (local file).
Next, I validate the data on the basis of columns present in the data.
Then data goes for preprocessing step, during this step data is being clean and all the required feature has been extracted from it using Polars library.

Model Overview

As you know I am working with videos title and tags which are textual data so I've used TfidfVectorizer (for text to vector conversion). I've used two TfidfVectorizer for each column (title and tags) and then used ColumnTransformer to create a (sort of) chain transformation step.

def get_vectorizer() -> ColumnTransformer:
    title_transformer = TfidfVectorizer(
        max_features=7000,
        ngram_range=(1, 2),
        preprocessor=preprocess_title,  # Preprocess texts of `title` column.
        stop_words="english",
    )
    tags_transformer = TfidfVectorizer(
        max_features=5000,
        ngram_range=(1, 2),
        preprocessor=preprocess_tags,  # Preprocess texts of `tags` column.
        stop_words="english",
    )
    transformer = ColumnTransformer(
        [
            ("title_trf", title_transformer, "title"),
            ("tags_trf", tags_transformer, "tags"),
        ]
    )
    return transformer

Data to Export

Now, I've successfully built the pipeline and trained the system but there comes a question that how to recommend a channel and for that I've to export some essential data like the vectorized array (vectorized videos titles and tags) with its metadata like channelId and channelTitle. To tackle this thing I've combine these data and created a pl.DataFrame and then export it as parquet format.

def training(
    input_data: Literal["db", "file"],
):
    # Extra code hidden...

    df = preprocess_data(df)
    transformer = get_vectorizer()
    transformed_data = transformer.fit_transform(df.to_pandas())

    # Combine transformed_data, channelId, channelTitle as DataFrame
    title_tags_trf_df = df.select("channelId", "channelTitle").with_columns(
        pl.lit(transformed_data.toarray()).alias("transformed_data")  # type: ignore
    )

    dump_object(transformer, CH_RECO_TRANSFORMER_PATH)  # Export `ColumnTransformer` object.
    # Export dataframe as parquet format for lesser size
    title_tags_trf_df.write_parquet(CH_RECO_TRANSFORMER_DATA_PATH)

[!IMPORTANT] What is parquet format?

Parquet is a columnar storage format that provides compression benefits and is particularly suitable for analytical queries.

Prediction Pipeline

I'm calling this step as Prediction Pipeline 🙂 because it doesn't feels good to call Reccommendation Pipeline 😞.

Here, I've to get any channel's data (videos titles and tags) to transform using stored ColumnTransformer object. After, transforming the data I've calculated cosine_similarity between new channel's vector and vector which I have stored on training step and from that whichever channel has greater similarity value is being reccommended to the user 🤩.

def prediction(data: pl.DataFrame):
    # Extra code hidden...

    transformer: ColumnTransformer = load_object(CH_RECO_TRANSFORMER_PATH)
    transformer_data = pl.read_parquet(CH_RECO_TRANSFORMER_DATA_PATH)

    transformed_data = transformer.transform(data.to_pandas())
    similarity = cosine_similarity(
        np.array(transformer_data["transformed_data"].to_list()),
        transformed_data.toarray(),  # type: ignore
    )
    return transformer_data.select("channelId", "channelTitle").with_columns(
        pl.lit(np.ravel(similarity)).alias("similarity"),
    )

Extra

[!NOTE] Recommendation System Summary

Ingesting data from database or local file. I had made API endpoint to fetch data from datbase.

Using Polars library for data manipulation.

This recommender system trained on YouTube Channel's Videos titles and tags which means it recommend on the basis of the channel's videos contents like title and tags.

Used TfidfVectorizer for text-to-vec conversion.

Provide Weights to Vectorizer

Previously, I thought that I can add a functionality to provide weights to each vectorizer (TfidfVectorizer) to make the system more robust and I had achieved it (See Notebook) but not feels good while actual implementation because it creates so much objects to store and makes the prediction (recommendation) step complex.

I have to store each vectorizer, vectorized data (title and tags) and the metadata (channelId and channelTitle) too which this pipeline complex and hard to keep track of objects.

Adding more Features

I have tried to add more features like categoryName (channel owner provide category of the video while uploading) and contentTypePred (a feature I have predicted using ML) but I found it difficult to implement and it doesn't show much affect while recommending. That's why I thought a different idea to implement this.

I can filter the recommended channels on the basis of categoryName and contentTypePred in the frontend part (yeah this not the right way of doing this but I'll think about it later).

🙏 Thank You for reading this. I am Anshul Raj Verma and I am a Data Scientist.

Channel Recommender System V2

Anshul Raj Verma — Sun, 21 Jan 2024 00:00:00 GMT

The system starts with preprocessing the video titles and tags by tokenizing, removing stop words, and stemming/lemmatizing them. Then, these texts are vectorized using TF-IDF to convert them into numerical form. The feature vectors are used to calculate similarities between channels using cosine similarity. The channels are ranked based on these similarity scores, and the top-N channels are recommended to the user.

This approach leverages the content-based filtering technique to find and recommend channels that have similar video titles and tags to those the user has already watched, ensuring relevant and personalized recommendations.

Approach Layers

Input Layer: Accepts video titles and tags.
Text Processing Layer: Applies tokenization, stop words removal, and stemming/lemmatization.
Vectorization Layer: Converts processed text to TF-IDF vectors.
Similarity Calculation Layer: Computes similarities between channels using cosine similarity.
Recommendation Layer: Ranks channels based on similarity scores and recommends the top-N channels.

graph
    subgraph DataPreprocessing
        C(Tokenization) --> D(Stop Words Removal)
        D --> E(Stemming/Lemmatization)
        E --> F(TF-IDF Vectorization)
    end

    subgraph SimilarityCalculation
        G(Feature Vectors) --> H(Similarity Calculation)
        H --> I(Channel Recommendations)
    end

    A1(Video Title) --> DataPreprocessing
    A2(Video Tags) --> DataPreprocessing
    DataPreprocessing --> pca{{PCA}} --> SimilarityCalculation

Process Explanation

Data Preprocessing:
- Tokenization: Split titles and tags into individual words or phrases.
- Stop Words Removal: Remove common words that do not add value to the recommendation (e.g., "the", "and").
- Stemming/Lemmatization: Reduce words to their base form to handle different variations.
- Vectorization: Convert the processed text into numerical vectors using methods like TF-IDF (Term Frequency-Inverse Document Frequency).
Feature Extraction:
- TF-IDF Vectorizer: Transform the textual data into feature vectors.
- Dimensionality Reduction: Optionally, use techniques like PCA (Principal Component Analysis) to reduce the feature space.
Model Training:
- Content-Based Filtering: Calculate the similarity between videos using cosine similarity or other distance metrics.
- Training Algorithm: Use algorithms like k-Nearest Neighbors (k-NN) for finding similar channels based on the content similarity of their videos.
Recommendation:
- For a given user's watched history, calculate the similarity scores of channels and recommend the top-N channels with the highest scores.

Dev Journal

Anshul Raj Verma — Mon, 08 Jan 2024 09:44:00 GMT

I came to know about Dev Journal from this YouTube video.

My Thought

It is a very easy and good way to review yourself on weekly basis. Using this you can keep track of your learning and improvement.

Convention

Using this I will be able to follow a set of instructions which helps me to write my Dev Journal in a pre-defined manner.

I have wrote snippets for this for vscode.

Monthly journal file starts with # Journal of <month-name>.
Journal will'be maintain on weekly basis.
Weekly Journal titled as ## Week <week-num> Journal.
Weekly Section doesn't contains any other sub-section.

Tips (using `ollama`)

Focus on the tasks that you completed during the week and what you learned from them.
Don't be afraid to include any mistakes or errors that you made during the week. It helps to learn and improve from them.
Use this as an opportunity to reflect on your progress and identify areas for improvement.
Keep your journal entries concise and focused, around one page or less.
Consider sharing your journal entries with a mentor or colleague for feedback and support.

Citations

Travis Media Video: Inspired from this video.
Advice From a Software Engineer With 8 Years of Experience | by Benoit Ruiz: The article discuss in the video.
Work log template for Software Engineers (The Pragmatic Engineer): The Google Doc in which the article writer written his Dev Journal. (Maybe this doc is for reference)
Templates as Inspiration for Software Engineers and Engineering Managers - The Pragmatic Engineer: Extra content for this thing.

January Journal

Anshul Raj Verma — Mon, 01 Jan 2024 00:00:00 GMT

Week 1: Dev Journal of January

Learning

ruff: An amazing python code linter and formatter with very high performance, written in Rust.
- Ruff GitHub: https://github.com/astral-sh/ruff
Docker: Containerise your application.
- You can containerise multiple apps using docker-compose.
- There are many concepts in docker I have learned Docker Volumes, Port Expose, Environment Variables Initialisation. Left out concepts: Docker Networking, Docker Hub, etc.
pyproject.toml: New way to configure you Python project.
- I used this to configure my ruff configs.
- I have not completely learned this configs management file exclusively regarding python projects.
Because this is the first week of the year, I have reviewed my configuration files from GitHub repository and GitHub Gist.

Thoughts

Ruff is amazing because it combines both linting and formatting together with its amazing performance speed ⚡.
Docker makes your application very easy to setup for other developer.
Learning MLOps is very iterative, disgusting, and many more. But when you learn it you will be like "WOW! What a service, product, concept."

Week 2: Dev Journal of January

Learning

VSCode UI Settings: Explore settings in VS Code IDE which will modify its UI colour (mainly).
Discussed Cold Turkey Blocker issue on Windows: Windows have the path related issue. Repository | GH2
Aggregation Pipeline in MongoDB: Used aggregation pipeline in my project which makes it easy to deal with data. I don't explicitly merging the datasets in the script.
- Commit where I did this in my yt-watch-history project.

Thoughts

Creating ML system is very tedious. I am trying to understand how to integrate these amazing tools in this. ALthough, I am able to train and do prediction with my ML Model but I am not satisfied with my approach. Thats why I am learning new ways to design these systems.

Week 3: Dev Journal of January

Learning

mkdocs-material: I have this amazing documentation maker using .md files in 1 day (almost). You can see my personal website hosted with GitHub Pages.
Freelance: @theabhinav002 queried to create #web-scrapping APIs for real estate website.
- I have accepted the offer.
- Now, I am creating the web-scrapper for housing.com.
yt-watch-history: I have been working around Channel Recommendation System which recommend channels on the basis of channel's videos title and tags. However, I have completely build the model architecture but I wonder to put it in a pipeline to make it more concise to use and define.
- I have discussed this system with @arpitpatel01.

Thoughts

Always write your thoughts whichever things you are working on because it helps you to understand your thoughts and also helps you that how you explain your things or projects.
While my freelancing process I feel like I am at my full potential and I've almost completed the project in 3-4 days. I have written code to scrape housing.com. Also created APIs around it using FastAPI framework.

Week 4: Dev Journal of January

Learning

yt-watch-history: This week I spend all the time on refactoring and understanding my project. It feels very difficult to work on but I made my mind and say that here I can push and pass my limits.
Custom Transformers using sklearn API: I have not totally learned how to create a custom transformer using sklearn API but I got the idea of it how to use it. Also, I have wrote some custom classes and tested it (it works well).
New Realease: I have released v0.0.1 of yt-watch-history project.

Thoughts

You can create your custom transformer using sklearn and then put them into a pipeline which works like magic.
I am going to break my limit with this project (yt-watch-history) because in this I am using too many new concepts like Docker, MLFlow, Custom sklearn transformers (recently). Now, I am thinking about AWS 🤩.

Week 5: Dev Journal of January

Learning

learnwith.campusx.in: I have written a script which goes through the html of the website and using the cookies it fetches the resources which were provided in the videos' description.
Learning ML Concepts: I have learned how Gini Impurity/Index works by practicing it on paper with my hand and pencil. This helps me a lot and I (sort of) figured out, how should I learn things in ML.
Blog on Decision Tree: I published a blog on decision tree on my website.

Thoughts

Frustration: In these few days, I am not feeling that inspiration while doing anything. I feel frustrated and difficult to focus on things. It is like something which I can't express but it is bad for me.

MLOps session by Pranjal Sir

Anshul Raj Verma — Mon, 27 Nov 2023 08:24:00 GMT

On 27 November, 2023 I am learning MLOps with CampusX DSMP 2.0 Course from @PranjalSir.

He introduced us MLOps in a very interesting way with a analogy of Organising a Concert for World Cup Final. You can read this notebook to see that beautiful analogy to get the high level idea about MLOps.

In MLOps, you have to learn many different tools to manage your codes, data, ML models and to visualize the model performance and many different things. Tools like Git, DVC, CookieCutter, DataBases, Cloud Services, Jira, Prometheus, Grafana, etc.

I met Natrajan Sir

Anshul Raj Verma — Fri, 27 Oct 2023 13:31:00 GMT

Today I realize that I again met another great guy @NatarajanLalgudi. He is just a amazing person. His style of talking, way of understanding and his opinion on a data is just interesting and amazing.

He teaches me many things in just few days such as document your projects and share it with other which works as marketing of yourself and I follow this advice and this works too. I learned a lot things from him.

We met while exchanging Real Estate Datasets which I scrapped from 99acres.com he said like you scrapped this amazing data, you are amazing; you have to document this scrapping project and share with others also upload this data on kaggle too with a proper documentation. So I did and it works well.

Today we connect on g-meet and he shows his works on the dataset and it looks fabulous I was just stun. He also suggest me to market yourself by sharing your knowledge and these yours awesome projects and also participate in Kaggle competitions which enhance your knowledge and it reflects on your CV too.

So I decided that I am going to move my focus from GitHub streak to Kaggle and Data Science concepts.

Thank You<br> Natarajan Sir

ARV's Blog

July Journal

Week 27 Journal

June Journal

Week 23 Journal

Week 24 Journal

Week 25 Journal

Week 26 Journal

AI Harness

Requirements

Agent Skills

AGENTS.md

Plan Mode

Open Source

Models Availability

Extensibility and Customization

May Journal

Week 19 Journal

Week 20 Journal

Week 21 Journal

Week 22 Journal

Goa Trip

24th April

25th April

26th April

27th April

28th April

How to explain a face from your thoughts?

April Journal

Week 14 Journal

Week 15 Journal

Week 16 Journal

Week 17 Journal

Week 18 Journal

96 by C. Prem Kumar and Govind Vasantha

Dehradun Trip

I Shouldn't Go

In the Trip

Day 01: Awesome Chhota

Birthday Dairy

Laptop

Spray Can

Day 02: New City, New Experience

Day 03: Ride up to Mussoorie

Day 04: Last Ride to Rishikesh

Day 05: Trip Hangover

Miscellaneous

An Expense Tracker

A Chatting Group

Backpack

Streaming Mode

Testing OBS Studio

Something to Learn with Code

Live Streaming

March Journal

Week 10 Journal

Week 11 Journal

Week 12 Journal

Week 13 Journal

Sit Down and Ask

SSH with Bitwarden

Manage SSH with Bitwarden

How Does Above Config Better?

Host And HostName In ~/.ssh/config

February Journal

Week 06 Journal

Week 07 Journal

Week 08 Journal

Week 09 Journal

The feeling at Platform No. 4

Docs Conventions

So What Are Docs Conventions?

Dotfiles Management

Migrate Docs to Fumadocs

My Requirements for the Docs

Previous Migration - diary/v1.0.0

Fumadocs Learning

File Structure

Define New Collections

Integrate Social Cards

`Host` And `HostName` In `~/.ssh/config`

Previous Migration - `diary/v1.0.0`