Why do we need to implement Master and Slave concept in simple words:

Suppose if we create any job in jenkins dashboard, by default we will execute that pipeline in jenkins CI/CD server,

By default, we can run 2 pipelines simultaneously on the Jenkins server. how we can say we can only run 2 pipelines at a time on jenkins server? if you observe keenly, in jenkins dashboard, you can check the no. of executors is 2, these executors can helps us to execute the pipelines.

If we increase executors to more than 10, there will be performance issues on the server, depending on the instance type. For example, if there are 2 vCPUs and 1 GB of RAM in a t3 micro or similar, increasing executors further may cause the server to crash due to performance issues. Because the server size is small and the pipelines are large, the server can crash. For that reason, we need to implement the master and slave server concept.

What exactly is Master and Slave?

suppose we have 10 pipelines, if we execute 10 pipelines at a time on a single server, then server may crash due to max load on single server. for that reason, we will create 5 Slave Servers .

Note: By default, there will be 2 executors in each pipeline.

So, we will create 5 slave servers, each slave server contains 2 pipelines, allowing all 10 pipelines to be executed across 5 different slave servers. This way, no maximum load will be placed on the single server, as the load will be distributed across each slave server. at a time, 10 pipelines are executed now. but here is the note, we will not install jenkins on each slave server. Jenkins is installed only on the master server. and here, we need to connect each slave server with master server. each master and slave server should have ssh connection.

due to this master and slave server connection, we can run any pipeline on any slave server.

The master server is the one where Jenkins is installed, and the slave server refers to the worker node that performs tasks assigned by the master node. The master server assigns tasks to the slave servers and manages them, essentially acting as their manager.

Master and Slave servers In Practical way

How do we know in which server our pipeline is running?

if it is showing default path of jenkins like this /var/lib/jenkins/workspace/ in console output**.** then we do know that, that pipeline is running on master server.

To connect slave server with the master server, then we need to install jenkins dependency that is Java 21 on slave server.

the command to install java 21 is yum install java-21-amazon-corretto -y

To check whether java is installed or not, the command is java -version

How to connect slave server with master server in practical guide

in manage jenkins in jenkins dashboard, there will be a popup displaying like setup agent, click on it, or else you can also go through nodes in manage jenkins and click on new node

if we execute any pipeline on master server, then workspace will be created on default storage path, that is /var/lib/jenkins/

but if we execute any pipeline on slave server, then workspace will be created on this path, /home/ec2-user/myjenkins/ but if you observe, myjenkins does not exist, but it will be automatically created.

if we try to remove it,

How to bring executors to online?

execute the below commands:

sudo mkdir -p /var/tmp_disk

sudo chmod 1777 /var/tmp_disk

sudo mount --bind /var/tmp_disk /tmp

echo '/var/tmp_disk /tmp none bind 0 0' | sudo tee -a /etc/fstab

sudo systemctl mask tmp.mount

How do we run the same job that was running on the master server on the slave server?

You need to assign the same matching label that was given to the slave server. we have previously checked the label as build should happen only when matching this label.

Note: to execute the pipeline on slave server, you need to install git on slave server.

Build & Unit Test

add maven in manage jenkins tools, and integrate maven in build steps of your job

Deployment

install tomcat in appserver and install deploy to container plugin in jenkins, then in post build actions integrate it.

How to manage your Tomcat now

Stop Tomcat

/root/apache-tomcat-9.0.115/bin/shutdown.sh

Start Tomcat

/root/apache-tomcat-9.0.115/bin/startup.sh

Code (Developer writes the code on their local system) => GitHub (once the code is ready, they push it to the central repository) => CI/CD(with that code, need to implement CI/CD to bring source code, and build and unit test), Application Server such as TOMCAT, JBOSS, GLASS FISH, WEB SPHERE (used to run the applications)

We need two servers here, one for CI/CD to retrieve the source code from Git, then build and test it, and another server is application server to run the applications.

Behind the Scenes of CI/CD Automation

Developers job is to write source code and push it to git, then devops engineer will create CI/CD Server to create the pipeline which is used retreive that source code from github and also to perform build and unittest the code. here jenkins is a CI/CD tool which is used for automation of this complete process.

There is a tool called Maven in Jenkins, which is used to perform build and unit tests on the code. Through Maven, there is a command that automatically compiles the code, runs unit tests, and packages the app, generating a WAR file (for java app).

WAR file contains the application code and its dependencies. If we place the WAR file on the application server, our application will be deployed.

How to enable Maven in jenkins?

Maven is a built-in default plugin in Jenkins. To enable Maven, go to Manage Jenkins => Tools => Maven Installations => Add Maven => given any name for maven then save it.

If you don't see the Maven version in Add Maven, go to https://downloads.apache.org/maven/ and check the version. Use the latest version.

Then go to Jenkins dashboard => your Jenkins job => configure => Build Steps => Source Code Management. If Source Code Management is already done, go to Build Steps => Add build steps => Invoke top-level Maven targets => select Maven version => Goals: clean package (clean package means it will delete any previous WAR files and start code compilation, perform unit tests, and package the app freshly so that new war file will be generated).

How to bring executors back online?

If disk space is below the threshold and there isn't enough space left, your node executors will go offline. How can you fix it?

execute the following commands:

sudo mkdir -p /var/tmp_disk

sudo chmod 1777 /var/tmp_disk

once after executing the following commands then need to restart the jenkins dashboard and check the executors status.

command to install Git: yum install git -y

How to check whether the build and unit tests were done successfully with Maven?

Go to the workspace in your job and check if the target folder is there. If you click on the target folder, the WAR file will be present.

whenever you have done the maven build, then target folder will be created automatically.

Create seperate server for running application server

Note: whatever security group you have taken for jenkins, the same security group need to taken for tomcat also.

yum install java-17-amazon-corretto -y wget https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.115/bin/apache-tomcat-9.0.115.tar.gz tar -zxvf apache-tomcat-9.0.115.tar.gz sed -i '56 a' apache-tomcat-9.0.115/conf/tomcat-users.xml sed -i '57 a' apache-tomcat-9.0.115/conf/tomcat-users.xml sed -i '58 a' apache-tomcat-9.0.115/conf/tomcat-users.xml sed -i '59 a' apache-tomcat-9.0.115/conf/tomcat-users.xml sed -i '56d' apache-tomcat-9.0.115/conf/tomcat-users.xml sed -i '21d' apache-tomcat-9.0.115/webapps/manager/META-INF/context.xml sed -i '22d' apache-tomcat-9.0.115/webapps/manager/META-INF/context.xml sh apache-tomcat-9.0.115/bin/startup.sh

the above script you need to install in your application server.

note: to check tomcat version is available or not, go to this site, https://downloads.apache.org/

and then check tomcat version if it matches with tomcat version which was in above mentioned script.

How to access the tomcat dashboard?

Along with the server IP and port 8080, for example: 54.242.234.24:8080

Click on manager app in tomcat dashboard => signin => for username and password, check the script you have executed previously, it will be there in tomcat script. you can search for username and password in above tomcat installation script. username="tomcat" password="admin123

Now the next step is to retrieve the app server packaged file (the war file) into the Tomcat server. for that, you need to install one plugin.

How to install Plugin in jenkins?

go to Manage jenkins => plugins => Available plugins => search for deploy to container plugin and then select and install it.

its time for deployment, so you need to select post build actions in configure, follow the steps:

now refresh the tomcat base url, then check the project by clicking on project name

Cron Syntax:

It has 5 starts * * * * *

1st star represents - no. of minutes

2nd star represents - Hours

3rd star represents - Day in a month (Date)

4th star represents - Month in a year

5th star represents - Day in a week

suppose you want your pipeline to be build at some scheduled time, this is the format of cron jon syntax, you should follow to write the cron job for suppose march 14, 2026 date at around evening 5:26

Note: sunday will start from 00 in cron job syntax

Cron job for the above time: 26 17 14 03 06

Note: Jenkins will not follow IST

How to change timezone of jenkins ?

Go to jenkins settings => Preferences => timezone => for example IST

Note: After updating the timezone, you need to restart the jenkins server and login again.

How to schedule cron job in jenkins?

Triggers => Build Periodically => Schedule => write cronjob

To build job for every minute: * * * * *

To build job for every 2 minutes: */2 * * * *

To build job for every 2 hours: * */2 * * *

Poll SCM:

By using pollscm also, we can schedule to build the jobs.

• * 9-17 * * * (for every minute in those duration of 9am to 5pm if there's any change in code, it's going to trigger the pipeline)

Difference between poll scm and build periodically?

Build periodically will execute the build because it is scheduled; it is not linked with the code.

Poll SCM will trigger the pipeline only when a developer makes any changes during the scheduled time, for example, 9-17., it is completely depends upon developers commits.

How to delete old builds from dashboard?

To discard Old Builds in jenkins dashboard, Go to jenkins dashboard => configure => General => click on discard old builds option,

give the no of days to keep builds, for example 3 days

max no. of builds to keep, top 5 builds

Throttle builds:

Throttle builds means that when you build one pipeline, you can only start the next pipeline after a certain amount of time has passed since the first build. For example, you might wait 5 minutes after completing the first build before starting the second one.

This will be useful to minimize the multiple builds at a time when 2 or more developers have a access to that pipeline and to avoid the server crash. basically throttle build is a queue process of pipeline builds.

Build after other projects are built:

Let's suppose we have four pipelines for each build: one for code, one for build, one for testing, and another for deployment. It will build a pipeline only after the previous one succeeds.

We can decide whether it can proceed to the next build even if it fails, based on the requirement.

Build Types:

1. Poll scm

2. webhooks

3. build periodically

4. throttlebuilds

5. Build after other projects are built

Job: Job is a task we are going to perform in jenkins.

There are different types of jobs in jenkins:

1. freestyle

2. pipeline

3. multibranch-pipeline, etc…

Source Code Management

here, we are going to select the tool you are using for source code management.

Note: we must use https while providing github url, Jenkins will not supports to SSH.

1. Repository URL: here you need to add your project github repo url.

2. Credentials: we don’t need to provide credentials if it is public repo as we know it.

3. Branches to build: add the branch where your project code exists to build.

After adding all the required data in the Jenkins task board, if you build the source code now, all your code will go from the repo to the CI server, once it is done, it will show you the indicator either sucess of fail.

you can check in workspace tab, whether your code is there or not after successful task build.

you can check the code after successfull completion of task in 2 ways:

1. Jenkins UI

2. in your server by going into the default path of jenkins

Default path of Jenkins:

/var/lib/jenkins

whenever you perform any task, that task info will be stored in this path only /var/lib/jenkins/workspace

Step1:

To Generate the token in Github for private repo:

Click on GIthub Profile → Settings → Developer Settings → Personal Access Tokens → Select the Class Tokens →Generate new token (Classic)

Select repo & then generate token

Step2:

you need to add credentials of private repo in jenkins

Select Add credentials => username with password => ID (ex: github-token) => ID (github username) Desc (github password)

Now build the pipelines to send all the GitHub changes to the CI Server. Note that this is a manual process of building pipelines because every time you need to build the pipeline to send GitHub changes to the CI Server. To automate the process, you need to implement webhooks so that whenever developer pushes those changes, the webhooks will trigger, and the pipeline builds to automate the process.

it may be either webhooks, pollscm, or build periodically to automate the build process.

Integration of Webhooks:

Step 1: Adding webhook steps in github

Github Repo Settings => Webhooks => Add webhook => in payload url, you need to provide your jenkins server url like in this format http://54.60.12.18:8080/github-webhook/ => application format should be in json format => add webhook. once it is added, then you can refresh the page and check, webhook was added successfully or not'

Step 2: Configure webhook in job of jenkins

Click on configure in jenkins dashboard => triggers => check this option “GitHub hook trigger for GITScm polling” (only if you select this option, then only webhook will be activate from jenkins job also)

Note: if any pipeline builds automatically without human interaction, then we can call it as trigger in technical terms.

Scenarios of pipeline failures:

To edit the job, click on configure

1. wrong Repo Url

2. wrong branch name

3. wrong credentials

4. check the error or stderror message in console output logs

5. check the token expired or not

2. Kernel → controls CPU, memory, disk, network. The brain of the system.

3. Operating System → kernel + tools + libraries + apps.

4. Distribution (Distro) → Linux kernel bundled with tools. Example: Ubuntu, CentOS.

5. GNU Tools → basic commands like ls, cp, grep. This is why people say GNU/Linux.

6. Terminal → where you type commands.

7. Shell → program that reads your commands. Bash is common.

8. File System → everything starts from /. No drive letters.

9. Root User → admin user with full control.

10. Permissions → decide who can read, write, or run files.

11. Process → a running program.

12. Service → program running in background.

13. Package Manager → installs software. apt, yum, dnf.

14. SSH → connect to another machine securely.

15. Logs → files that record system activity and errors.

Continuous Integration:

continuos integration is nothing but both build and test happens at same time, it is combination of continuos build and test.

Before CI, imagine one developer wrote code, pushed it to GitHub, and then another developer would then clone the codebase and install the required app dependencies and build or compile the code manually and package the app. Then the QA team would execute unit tests. If any issue was found during testing, the code was sent back to developer one for fixing. Once they fixed it, the same cycle would be repeated, such as rebuilding, retesting, and repackaging. All this was a manual process, involving repetition of the same tasks. Here, the entire process was done manually, and it was time-consuming as if app size grew.

After CI, developers write code and push it to GitHub. Whenever code is pushed to GitHub, CI server will be automatically triggers through webhook. CI Server then automatically build the application, and executes automated tests. At this point, both build and test are automatically performed at the same time. If the test fails, developers fix it and CI server will be retriggers automatically, where the CI server will automatically perform the rebuild and retest. Here, the entire process is done through automation.

Continuos Delivery:

Continuos delivery is nothing but manual deployment process.

Continuous delivery means the application is always kept ready for release. The code is automatically built, tested, and prepared through pipelines, but the final deployment to production is done manually when the team decides to release.

Continuos Deployment:

Continuos delivery is nothing but automation deployment process.

Continuous deployment goes one step further. After the code passes all automated tests, it is automatically deployed to production without manual approval. Every successful change can reach users quickly through full automation.

What is Pipeline?

Pipeline is used to automate the process of code, build, test and deployment stages, we need a tool to create the pipeline to automate the deployment process. Jenkins, Gitlab, Github actions, AWS Code pipelines, Travis CI, Circle CI, Harness, Teamcity are such CI/CD tools which is used to manage the pipelines.

What is Jenkins?

Jenkins is a CI/CD tool which is used to build and manage the pipelines to automate the Software Development Life Cycle(SDLC).

Jenkins follows Master-Slave Architecture. it is the main feature in jenkins tool.

Master & Slave Concept explanation in simple way:

Here, master and slave are servers. Think of the master server as a manager who assigns work to team members, and the slave servers as the ones who actually do the work. Whatever tasks they receive, they will complete. Without the master and slave server concept, if we run all our applications on one server, it may crash due to heavy load. Instead, we can divide it into two servers: the master server (which manages all slave servers) and the slave server (which performs the tasks). Each slave server is responsible for running the pipeline assigned to it, but managing the pipeline is the responsibility of the master server. master server is where Jenkins runs.

We will use the master and slave server concept to spread the server load across different slave server pipelines as the application grows and has many services to run.

Connection Guide of Jenkins:

8080 is the default Jenkins port.

To get access of jenkins dashboard, you can access with your server public ip and jenkins port, example: public-ip:8080.

Suppose your server is in AWS EC2 and it has:

• Private IP → 172.31.40.25

• Public IP → 13.233.120.55

Important Note (Security Group):

When you request the Jenkins dashboard URL:

Before the request reaches the server, it first passes through the Amazon Web Services (AWS) Security Group attached to the EC2 instance.

The Security Group checks:

• Whether port 8080 is allowed in the Inbound Rules

• Whether your IP address is allowed to access it

If port 8080 is allowed, then the server gives access to the Jenkins dashboard.

If port 8080 is NOT allowed, the browser will show:

• Timeout error

• Site can't be reached

This is the flow of how url request to the server:

Browser → Internet → Security Group → EC2 → Jenkins → Response

You need to give 8080 port in inbound rules of aws security group, to get access to jenkins dashboard.

What is TCP?

TCP stands for Transmission Control Protocol. It is a communication protocol that allows data to travel safe and sceurely through internet.

Commands to install Jenkins:

you can create any file and copy the jenkins script but filename should be ends with .sh represents scripting file.

if not file, you can directly execute the script without adding the script in file.

Execute Jenkins file:

sh jenkins.sh: Here, jenkins is the filename, you can use any name you want.

systemctl status jenkins: Command to Check whether jenkins is in running state or not

systemctl start jenkins: Command to start the jenkins server

systemctl stop jenkins: Command to stop the jenkins server

systemctl restart jenkins: Command to restart the jenkins server

In simple terms, Git branch is like making a copy of the current working code, which we can call v1. Instead of changing the original code directly in v1 on the same branch, we create a new branch from the existing one, and all the existing code is automatically copied to the new branch. You can then modify the code in this copied version as needed. If everything works as expected after testing, you can deploy this version as v2 in the new branch. Finally, you merge this new branch with the existing one. This is the concept of branch workflow.

• By default master is the default branch in git.

Listing Branches

git branch: to see all the list of branches.

Note: If we don't make an initial commit, then default branch will not be shown.

Note: whenever we create a new branch from existing branch, then only committed files will be cloned in new branch.

Create new branch

git branch branchname: to create the new branch.

Switching the branch

git checkout branchname: to switch any specific branch

Merging branches

note: to merge any branch, you need to checkout to default branch that is master branch, then only you can merge.

note: you are creating files on server, so whatever file you created in any specific branch, same file will be replicated in all branches irrespective of on which branch, you’ve created it.

only once you commited that file, then that file will not replicate it on all branches, it will exist only on that specific branch.

git merge branchname: combines two branches and keeps all commits exactly as they were, adding one extra merge commit.

git rebase branchname: moves your current branch’s commits on top of another branch without creating a merge commit.

Difference between rebase and merge:
git merge keeps all commits and adds a merge commit by showing the branch history.
git rebase rewrites your commits on top of another branch, keeping history linear with no merge commit.

Create and Checkout branch in same command

git checkout -b branchname: to create new branch and checkout to that branch with same command.

Delete branch

Note: We cannot delete a branch while we are currently on that branch.

git branch -d branchname: to delete any branch

Difference between git branch -d branchname and git branch -D branchname:

git branch -d branchname safely deletes a branch only if it’s fully merged, while git branch -D branchname force deletes it even if it’s not merged.

what is the purpose of .gitignore file?

If we want to exclude specific files from being tracked and untracked, we can list those file names in the .gitignore file.

Git log:

git log: To get complete info about the commits we made.

git log -2: To get complete info about the last 2 commits we made.

git log —oneline: git log will return complete info about the commits we made. but Instead, we want to get only the half commitId and commit message, we can use this command.

git log —pretty=oneline: To get complete commitId and commit message, we can use this command.

How many commits have been made for a specific file? How can we find out? What is the command for this?

git log —follow —all filename: To get all the commits we made for this specific file. or we can also use git log filename but it will not be accurate if it has less commits.

Git show:

How many files have been committed for any commitId? How can we find out? What is the command for this?

git show commitId —name-only: To get only the filenames along with commit message.

git show commitId —stat: To get overall number of changes in the file, that is the total number of modification stats in that commit ID.

Git amend:

what does it mean amend in git?

--amend means: modify the most recent commit

git commit —amend —author “authorname <authoremail>”: To change the author for latest commit.

git commit —amend -m “authorname <authoremail>”: To change the commit message for latest commit.

If I committed 4 out of 5 tracked files and forgot to add the 5th tracked file in the same commit, but I need to include the 5th file in the same commit ID, and want all commits to be under the same commit ID, what is the command for this? In short, how to attach all tracked files in staging area to the latest commitId in this case?

git commit —amend —no-edit

How to change the last recent commit id message ?

git commit —amend

Git reset:

--hard means: move everything back and discard changes completely

git reset —hard HEAD~1: to delete the last commit.

in this command, HEAD represents latest commit, and HEAD~1 means last commit, if we give HEAD~2 it means, last 2 commits

On a final note, we are telling Git to discard the complete changes of the last specified number of commits. it will notify HEAD is now at below commit after delete of recent commit.

Git soft:

--soft means: only the commit will be deleted, but not the data, so the data is safe.

git reset —soft HEAD~1: to delete the last commit, but the file will not be deleted.

• Git represents Global Information tracker

• Git is basically a developer’s tool, in technical terms, Git is a free and open source Distributed version control system(VCS), and we can call it as Source code management(SCM) tool.

• Git was intoduced in year 2005.

• it will maintain multiple versions of same file.

• in simple way, Git tracks of every changes you made, so you have record of what has been done, and you can revert back to any specific version. it makes collaboration easy and allow changes done by multiple people and mege it into single source in final.

But what does Version Control System do in practically?

• In every application, there are multiple versions of code. For example, when an application is first created, it is called Version 1. As the application grows and new features are added, it is deployed with newer versions of the code, such as v2, v3, and so on.

• However, if after deploying a new version to production a feature breaks or the entire application crashes due to faulty code, this is where a Version Control System (VCS) becomes useful. Using a VCS, we can safely roll back from v4 to v3 or any version, the previous stable version in which the application worked correctly.

• In simple terms, a version control system allows us to revert to any earlier version of the code without any worry.

But what does tracking in git means?

• Tracking a file means it will record complete information about who created that file, when the last changes were made, when exactly it was modified, and what changes were made.

• so to keep track of those changes in file, we need tool called GIT.

Git Stages:

• entire git depends upon these 3 stages:

  1. Working directory

  2. staging area

  3. repository

• Working Directory:

  whatever the files created initially in any new project, it will comes under working directory, these files in working directory are not going to track by git,

• staging area:

  When we run git add ., the files are moved from the working directory to the staging area. In simple terms, the staging area is where developers make changes that is either data deletion, add or modify. Once changes are committed, they move from the staging area to the repository.

• repository:

  If we make changes to the same file again after committing, it will move from the repository back to the staging area.

why do we need to commit?

To keep track of the history of changes, that is “who made those changes”, “when they were made those changes”, and “what changes they have done”, we need to commit every time when the changes are made.

• To install git package in linux, the command is: yum install git -y

• To check version of installed git package, the command is: git -v or git - -version or rpm -qa git

initializing git in working directory:

.git: .git means a hidden folder, which is used to keep track of every changes we made, then it will record the history of the changes we made, once we commit it.

git init: to initialize the git in our working directory, we need to add git init to keep track all the files in our project.

git add filename: To keep track of changes we made in any file, we need to do git add filename, then it will be moved from the working directory to the staging area. Now we can commit those changes, and it will keep a history of the changes we have done.

git add * to keep track of all files and folders

git add . To keep track of all files, folders, and hidden files and folders

git commit -m “commit message” filename: To save the history of changes we've made, we need to commit those changes.

git log: To view the history of changes that have been committed.

git show commitId: To see the exact changes made in a specific commit and get complete information about who made those changes, why they made them, and when they did so, use git show commitId.

git config user.name “name”: to set the user name in git. (note: The new name will only appear after a new commit has been made. It won't show for previous commits.)

git config user.email “email”: to set the user email in git. (note: The new email will only appear after a new commit has been made. It won't show for previous commits.)

How to unstage or untrack a file after using the git add command?

git restore —staged filename or git rm —cached filename or git reset or git reset HEAD . To reset the changes that were added to the staging area, use this command.

There are different categories of Linux commands. In the previous chapter, we learned about data and editor commands. Now, in Chapter 6, we will dive deep into modify file owners and search commands. I will split the Linux commands into individual articles instead of covering everything in a single article.

6th chapter is modify file owners and search commands:

Changing owner and groupname:

1. chown user filename: to change the owner of the file with the username, that is ownername.

2. chgrp group filename: to change the group of the file with the groupname.

Full permissions to file:

1. chmod 777 fol1/file1 fol2/file2: giving full permissions to multiple files.

changing owner and group with single command:

1. chown user:group filenames: to change the owner and group for multiple files simultaneously.

2. chown user:group *: to change the owner and group for all files at a time.

3. chown user:group foldername: to change the owner and group for only folder.

4. chown user:group -R foldername: to change the owner and group for both folder and files.

5. chown user:group foldername/*: to change the owner and group for only files inside that folder.

6. chown user:group foldername/filename: to change the owner and group for only specific file inside that folder.

Search commands:

1. /param: to find the word in any file using vim editor.

2. /GREP: GREP stands for Global Regular Expression Print. which is used to search for a word in a file.

3. grep “word” filename: used to search word based on filename.

4. grep -n “word” filename: used to search word along with line number based on filename.

5. grep -in “word” filename: used to search for a word, case-insensitive, along with the line number in a file.

6. grep “word” filename: used to search word based on filename.

find commands based on file/folder:

1. find . -name filename/foldername: to find the path of the file or folder with that name in every directory if we give dot after find. (dot means current working directory)

2. find fol1 -name filename: to find the path of the file in fol1 directory if we specified directory name after find.

3. find . -type f -name filename: to find the path of the file in every directory

4. find . -type d -name filename: to find the path of the folder in every directory.

find commands based on file owners and file permissions:

1. find . -user username: to find all the path of files and folders based on file owner.

2. find . -type f -user username: to find all the path of files based on file owner.

3. find . -type d -user username: to find all the path of directories based on file owner.

find commands based on groups:

1. find . -group groupname: to find all the path of files and folders based on groupname.

2. find . -type f -group groupname: to find all the path of files based on groupname.

3. find . -type d -group groupname: to find all the path of folders based on groupname.

find commands based on permissions:

1. find . -perm 777: to find all the path of files and folders based on full permissions.

2. find . -type f -perm 777: to find all the path of files based on full permissions.

3. find . -type d -perm 777: to find all the path of folders based on full permissions.

Locate commands:

to use locate command, you need to install locate package with yum install locate -y and you need to use updatedb whenever you installed any package.

1. locate filename: to find the path of a file and folder

Difference between find, grep and locate commands:

locate filename: all 3 commands used to search the path of a file but there is a difference:

1. GREP command is used to search for the words in a file.

2. FIND and LOCATE commands are used to search for the file and folder.

3. FIND commands are used to search for the files and folders based on the path.

4. LOCATE commands are used to search for the files and folders in the entire linux database.

Software commands:

yum:

yum(yellowdog updater modifier) is a linux package management tool which is used to install, uninstall or update the packages.

yum install packageName -y: to install any package for example git i.e.. yum install git -y.

for redhat, yum install git -y

for ubuntu, apt install git -y

There are different categories of Linux commands. In the previous chapter, we learned about data and editor commands. Now, in Chapter 5, we will dive deep into user permissions and search commands. I will split the Linux commands into individual articles instead of covering everything in a single article.

5th chapter is User permissions and Search Commands:

as the name suggests, user and search commands in Linux are commands used to manage users and groups on the system.

They help you:

• Create or delete users

• Change passwords

• Switch between users

• Control user permissions and groups

Copy data:

1. cp: copy the complete data from one file to another file

2. cp sourcefile destinationfile: copy the complete data from source file to destination file by overwritting the previous data

3. cat >> filename: to append the data in the destination file

4. cat source >> destination: used to add source file data to destination file by overwritting the previous data.

   there are 2 commands in this, cat source means to read the sourcefile content, and then \>> destination means to append the new data in destination file without overwritting the previous data.

5. cat >> filename: to append the data in the destination file

Copy files in folder:

1. cp sourcefiles destinationfolder: used to copy all the source files and paste those files in destination folder. (will be acts as ctrl+ c and paste)

2. mv sourcefiles destinationfolder: used to cut all the source files and paste those files in destination folder. (will be acts as ctrl+ x and paste)

Rename files:

1. mv sourceName destName: rename the existing file with new name.

Move back directory to root:

1. mv fol2/fol1 ~/ :To move folder1 from folder2 back to the root directory, use the command: mv fol2/fol1 ~/.

copy file from existing directory to another directory:

1. cp folder1/file1 folder2: copy file1 from folder1 to folder2.

2. mv folder1/file1 folder2: move file1 from folder1 to folder2.

1. Permission commands:

1. -rw-r--r--

   - represents type of the file(TBF - text based file)

   d directory

   r read => 4

   w write => 2

   x execute => 1

   last - no permission => 0

   we have to divide complete rw-r--r-- into 3 parts:

   1. rw- (read & write) first 3 will be called as user permissions = read = 4, write = 2, - = 0 => total = 6

   2. r— (read only permissions) next 3 will be called as group permissions

      read - 4, - = 0 => total = 4

   3. r— (read only permissions) last 3 will be called as others

      read - 4, - = 0 => total = 4

total: rw-r--r-- = 644

2. whenever any system admin will see the permissions rw-r--r, then they will better understand it by decoding the value from it, that is total of 644, they can understand, user has read & write permissions, and group has read only permissions, and others has also read only permissions.

Change file permissions:

1. chmod: used to change the permissions that is change mode

   to give full permissions for all users, = 777 (r-4, w-2, x-1)

2. chmod 777 filename/ chmod 777 multifiles - to give complete permissions to complete file

3. chmod +x filename = to give execute permissions to complete file.

4. chmod 644 * = to give default permissions to all files

Change folder permissions:

1. chmod 777 foldername/ chmod 777 multifolders - to give complete permissions to complete file

Change Folder Permissions and Apply Them to Files Inside Simultaneously:

1. cmod 555 foldername -R: To change folder permissions, which also apply to the files inside the folder.

Change Permissions for Only the Files Inside a Folder:

1. cmod 777 foldername/*: To change permissions for all files inside folder1, without affecting the folder itself, use the following command. This will apply the changes only to the files inside.

2. User Commands:

1. cat /etc/passwd: to see list of users (-n to see numbered list)

2. getent passwd: to see list of users (numbered list will not work here)

Create Users:

1. adduser username or useradd username: to create new user

2. id username: to see if user is created or not

3. uid=1001(bhanu) gid=1001(bhanu) groups=1001(bhanu): When a new user is created, a new group is automatically created with the same user ID (uid) and group ID (gid).

4. when a new user is created, a new path is also created for that user with tha username.

   user => group => path

Delete Users:

1. userdel -r username: delete the user

navigate to user:

1. su - username: to navigate to that user (su = switch user)

set password for user:

1. passwd username: to add new password for that user

2. while switching from root user to any user, it will not ask password though you’ve set it, because if you are in root user, it has every access and every permission, that is the reason.

3. Group commands:

Why do we need a group?

for sepecific users, we can create a common group and give file permissions to those specific users.

1. cat /etc/group: to see list of groups(-n to see numbered list)

2. getent group: to see list of groups

create group:

1. groupadd groupname: to create a new group.

2. usermod -aG groupname username (modify user to add to that groupname): to add list of users to any specific group.

3. id username: To see if a specific user is in a group or to find out which groups a user belongs to.

delete user from that group:

1. sudo gpasswd -d dev devops: delete user from specific group. (sudo = super user do, gpasswd = manage group accounts, -d = delete)

There are different categories of Linux commands. In the previous chapter, we learned about file commands. Now, in Chapter 4, we will dive deep into Data and Editor commands. I will split the Linux commands into individual articles instead of covering everything in a single article.

4th chapter is data and editor commands:

as the name suggests, data commands in Linux are commands used to view, process, or manipulate data/content (like text or files).

editor commands in Linux are Commands used to create and edit files using text editors.

parenting folder creation:

1. mkdir -p folder1/folder2/folder3: creating folder3 directly inside other directories without creating them manually

2. cat > filename: to add the content in file by overwrite the previous data

3. cat >> filename: to append the new data in the file

4. cat filename: to read the file content.

5. cat -n a1: to see the data as numbered list in the file

Modifying Data:

1. Vim editor: used to modify any files on linux os. it has different modes:

   1. command mode

   2. insert mode

   3. save and quit mode

command mode:

it is default mode in vim editor, it is used to copy, paste and move to specific line, replace the words or search the words and for delete.

1. yy: used to copy the entire line and enter p to print the line

2. 4yy: copy multiple lines, copy 4 lines from where cursor is there

3. 10p: print the line 10 times

4. dd: delete the line

5. 9dd: delete the 9lines from where our cursor is.

6. G: to move the cursor to the last line.

7. gg: to move the cursor to the first line.

8. M: to move the cursor to the middle of the line.

9. 4gg: to move the cursor to the 4th line. or :4

10. G: to move the cursor to the last line.

11. u: undo all the changes.

12. CTRL + r: redo all the changes.

13. u: undo all the changes.

Search the word:

1. /: used to search for a word in file

2. **%s/old/new/g :**used to replace the old word with new word globally (s means string)

3. /: used to search for a word in file

Insert Mode:

used to insert the data in a file

1. i: used to go to insert mode

2. q: used to quit from vim editor.

3. :wq - save and quit

4. :wq! - save and quit forcefully

5. :w - save the data in file

6. :q! - forcefully quit without saving

7. A: to move the cursor to end of the line

8. I: to move the cursor to start of the line

There are different categories of Linux commands. In the previous chapter, we learned about hardware commands. Now, in Chapter 3, we will dive deep into file commands. I will split the Linux commands into individual articles instead of covering everything in a single article.

3rd chapter is file commands:

as the name suggests, File commands in Linux are Commands used to create, view, copy, move, rename, and delete files in Linux.

Create files:

1. touch: used to create file.

2. touch a b c: you can create multiple files at once

3. touch a{1..3}: sequential way of creating files like a1, a2, a3

4. touch a{1..3}.txt: sequential way of creating files like a1, a2, a3 with a common ext.

5. ll: to get full info. about file

6. ls: list of created files

Delete files:

1. rm : to remove the empty files

2. rm -f filename: to remove the file forcefully

3. rm -f *: remove all files forcefully

4. rm -f *.txt: removing all files with .txt extension

5. rm -f a{1..3}: sequential way of removing files like a1, a2, a3

6. rm -f a{1..3}.txt: sequential way of removing files like a1, a2, a3 with a common ext.

7. rm -f b*: remove files started with b letter forcefully

8. rm -f *: remove all files forcefully

Create Directories:

1. mkdir a: create directory named a

2. mkdir a b c: you can create multiple files at once

3. mkdir a{1..3}: sequential way of creating files like a1, a2, a3

Remove Directories:

1. rmdir : to remove the empty directories

2. rmdir *: to remove all the directories

3. rm -rf foldername: to remove non empty directories (rf - recursive forcefully)

4. rm -rf *: remove all non empty files and folders forcefully

5. rm -rf *: remove all non empty files and folders forcefully

Folder navigation:

1. cd: change directory

2. mdkir test: creating test directory

3. cd test: navigate to test dir

4. pwd: present working directory path

5. cd: to change the directory from pwd to root dir we can use this command

6. cd -: to change the directory from root dir to pwd dir we can use this command

7. cd .. to move one step back from pwd to previous directory

8. cd ../../ to move two steps back from pwd to previous directory

9. cd ../../../ to move three steps back from pwd to previous directory

10. cd fol1/fol2/fol3/fol4: to navigate to fol4 directly

file creation inside folder:

1. touch folder1/file1: creating file directly inside existing folder1

2. ll folder1/: check whether file is created or not inside that folder

3. ll folder1/fol2: creating another directory inside existing folder1

To create folder inside another folder (which never exists):

mkdir -p folder1/folder2: This command creates folder2 inside folder1, even if neither of these folders exists yet.

To create file inside another folder (which never exists):

mkdir -p newfolder && touch newfolder/newfile: create new file inside new folder.

listing files:

1. ll -r: listing files in reverse order

2. ll -t: listing recently created ordered files

3. ll -lrt/ll -rtl/ ll -tlr/ll -l: normal ordered listing

4. ll -a: to see hidden files (any files that starts with .)

Key Difference between folder and file creation in linux: file creation starts with “-”

in linux,

- if any file starts with -, it will be Regular file

d if any file starts with d, it will be directory

Understanding Linux File Directory System:

cd /: navigate to this / directory from root, and then perform ll for listing out the command to check complete Linux file directory system.

bin: This directory contains executable files. It includes files with full permissions.

boot: This directory contains the necessary files for booting.

dev: This directory contains device files related to hardware.

etc: This directory contains system config files (like jenkins or git or ansible…)

home: This directory contains all user related files.

lib: This directory contains shared library files that are required to boot the system.

media: This directory contains additional removable disk related files.

opt: This directory contains optional files.

proc: This directory contains processor related files i.e cpu and memory info...

root: This directory contains root directory files.

run: This directory contains necessary files to install packages.

sys: This directory contains system related files.

tmp: This directory contains temporary files.

usr: This directory contains application files.

var: This directory contains log information files for any installed tools.

There are different categories of Linux commands. In the previous chapter, we learned about system commands. Now, in Chapter 2, we will dive deep into hardware commands. I will split the Linux commands into individual articles instead of covering everything in a single article.

2nd chapter is all about hardware commands:

as the name suggests, Hardware commands in Linux are commands that help you check, control, or get information about your computer’s physical parts (hardware) using the terminal.

Ram and CPU info:

1. lscpu: to get the complete info about cpu

2. cat /proc/cpuinfo: command to get cpu info using cat command

3. top: to know about server cpu utilization.

4. ps -aux: command to see running processors on linux system.

5. free: to know how much ram our server consumed and also to know the available ram (in kb)

6. free -m or cat /proc/meminfo: to know how much ram our server consumed and also to know the available ram (in mb)

7. free -g: to know how much ram our server consumed and also to know the available ram (in gb)

8. free -h: to know how much ram our server consumed and also to know the available ram (in human readable format)

9. free -g: to know how much ram our server consumed and also to know the available ram (in gb)

Memory Info:

1. df -h: how much ebs volume consumed (to check the storage)

2. du -sh *: It displays the size of each file and folder in the current directory.

3. lsblk: to know about how many ebs volumes attached to our system (list of block devices)

There are different categories of Linux commands. to get started with system commands as Chapter 1, we will dive deep into system commands. I will split the Linux commands into individual articles instead of covering everything in a single article.

as the name suggests, system commands help you talk to the Linux OS and manage files, users, processes, memory, disks, and services.

1. sudo: the name sudo means super user and do in general

   it was used to switching from one user to another user.

   for example,

   sudo -i : by default, ec2 is the root user in linux os, but it has limited permissions, you need permissions to modify anything, ec2 has limited permissions. to switch from ec2 to root user, u need sudo -i

2. clear: used to clear the terminal or you can use alternative command with CTRL + l

3. exit: used to exit from root user or you can do exit with other alternative by using CTRL + D

4. uname: used to know which os we’re currently on. for example: linux

5. uname -r: used to know which kernel version our os is, for example: 6.1.159-181.297.amzn2023.x86_64

6. uname -a: used to get full info of our os

7. uptime: used to know how long os has been in running state

8. uptime -p: used to give only the time os has been in running state

9. hostname: used to get private dns name of our system

10. hostname -i: used to get private ip address of our system

11. hostnamectl set-hostname “bhanu”: used to change the existing hostname of our system

12. who: to know about how many users have been loggedin our system

13. whoami: used to see the current user

14. history: to listout the history of command we’ve used in that system

15. ip addr: used to get private ip adress

16. ip route: used to get private ip adress

17. ifconfig: used to get private ip adress

18. date: to get today’s date

19. date +”%d”: to get only date

20. date +”%m”: to get only month

21. date +”%y”: to get only year

22. date +”%H”: to get only hour

23. date +”%M”: to get only minutes

24. date +”%S”: to get only seconds

25. date +”%F”: to get full date in yy/mm/dd format

26. date +”%A”: to get month in year

27. timedatectl: used to get timezones

28. timedatectl set-timezone Asia/Kolkata : used to change Timezone to IST

29. ps: used to see the running processors in system

30. kill -9 pid: used to remove unwanted/unused processors

In simple terms, ec2 is used to create the virtual servers in the cloud

AWS provides 200+ services and ec2 is one of the service.

in devops terminology, to create instance(server), we need to use EC2 service

Think of EC2 as a computer (server) that lives on the internet, provided by Amazon Web Services.

Instead of:

• Buying a physical server 💻

• Keeping it in an office 🏢

• Managing power, cooling, hardware failures

• You can:

  • Create a virtual computer in minutes

  • Access it from anywhere

  • Pay only for the time you use it

  • EC2 = Renting a computer from AWS over the internet

Understand the difference between physical server and virtual server

8 Simple Steps to Create an EC2 Server:

Step 1: Create an AWS Account

Go to https://aws.amazon.com

Sign up using email, phone number, and card (free tier available).

Open EC2 Dashboard

Login to AWS Console

• Search EC2

• Click EC2 → Instances → Launch instance

• 

  Click “Launch Instance”

  

  Step 2: choose an AMI (amazon machine image)

  it is a template that contains software configuration (os, application server and applications) required to launch your instance

• Each AMI contains unique AMI-ID which is region-specific

• A region we call it as group of Data Centres, A Data Center we call it in technical terms as Availabilty Zones.

• 

Step 3: choose an Instance Type

in the 3rd step, we are providing cpu and memory to our instance, here we need to select the instance type

Step 4: Configure your instance

In this step, we are going to decide how many number of instances you can create from same AMI,

this step defines, in which network, and in which availabilty zone, you are going to launch your instance

Step 5: ADD Storage

To store the data in server, we need ebs volume.

EBS fullform is Elastic Block Storage

for single server, we can use multple EBS volumes, os can run on multiple EBS volumes

Step 6: ADD Tags

we can give a name to our instance

Step 7: Configure security group

Security group means group of firewalls which are attached to instance to control the traffic of an instance.

There are 2 types of traffic: 1. Inbound traffic 2. outbound traffic

Inbound traffic means security group which allows all the incoming requests with the ssh (22 is default linux port) for macos(remote desktop rdp)

if we want to connect to Linux system to launch the instance, then we need ssh(22) port in the security group

Step 8: Key-pair

we have to create an instance with key-pair: 1. public key 2. private key

public key is available in aws

private key is available in our system when we downloaded

To connect our linux system to launch an instance in a most safe and secured manner, we need key-pair, whenever public key matches with private key, then system is connected to the server.