1. Introduction

1.1 Tutorial Goal

This tutorial aims to provide a basic understanding of Hadoop and HDFS (Hadoop Distributed File System). We will learn to set up a Hadoop environment, understand the basic operations of HDFS, and write simple programs to interact with HDFS.

1.2 Learning Outcomes

By the end of this tutorial, you will be able to:

• Understand Hadoop and HDFS
• Set up a Hadoop environment
• Perform basic operations on HDFS
• Write programs to interact with HDFS

1.3 Prerequisites

Basic knowledge of Linux commands and Java programming is recommended, but not mandatory.

2. Step-by-Step Guide

2.1 Hadoop

Hadoop is an open-source software framework for storing and processing big data in a distributed manner on large clusters of commodity hardware. Essentially, it accomplishes two tasks: massive data storage and faster processing.

2.2 HDFS

HDFS (Hadoop Distributed File System) is the primary storage system used by Hadoop applications. It is a distributed file system that allows you to store data across multiple machines.

2.3 Hadoop Installation

You can download Hadoop from the official Apache website. After downloading, extract the zip file and set the environment variables for Hadoop in your bash profile.

# Unzip the downloaded file
tar xzf hadoop-3.3.0.tar.gz

# Set the environment variables
export HADOOP_HOME=/path/to/hadoop
export PATH=$PATH:$HADOOP_HOME/bin

3. Code Examples

3.1 HDFS Commands

HDFS provides shell-like commands to interact with HDFS directly.

3.1.1 List Files

# List files in the root directory
hadoop fs -ls /

# Expected output: A list of files/directories in the root of HDFS

3.1.2 Create Directory

# Create a directory named 'test'
hadoop fs -mkdir /test

# Expected output: No output if the command is successful

3.1.3 Upload File

# Upload a local file to HDFS
hadoop fs -put localfile.txt /test

# Expected output: No output if the command is successful

3.2 Java API

Hadoop provides a Java API to interact with HDFS programmatically.

3.2.1 Read a File

// Import necessary classes
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;

// Initialize the configuration
Configuration conf = new Configuration();

// Set the HDFS server
conf.set("fs.defaultFS", "hdfs://localhost:9000");

// Create a filesystem object
FileSystem fs = FileSystem.get(conf);

// Specify the file path
Path path = new Path("/test/localfile.txt");

// Open the file for reading
FSDataInputStream inStream = fs.open(path);

// Read the file
String line;
while ((line = inStream.readLine()) != null) {
    System.out.println(line);
}

// Close the file and filesystem
inStream.close();
fs.close();

4. Summary

We have covered the basics of Hadoop and HDFS, including how to install Hadoop, perform basic operations on HDFS using both shell commands and Java API.

5. Practice Exercises

1. Exercise 1: Use HDFS commands to create a new directory, upload a file to it, and list the files in the directory.

2. Exercise 2: Write a Java program to write data to a file in HDFS.

3. Exercise 3: Write a Java program that counts the number of lines in a file in HDFS.

6. Additional Resources

1. Hadoop official documentation
2. HDFS Java API documentation

Remember, practice is the key to mastering any skill, so be sure to practice these exercises and explore the additional resources to deepen your understanding of Hadoop and HDFS.