Thread Pools in Python

The ability to utilize multiple threads in parallel can significantly speed up the execution of computationally expensive tasks. However, managing multiple threads manually can be quite challenging and error-prone. That's where thread pools come into play. Thread pools allow us to efficiently manage and control a fixed number of threads that can be reused for multiple tasks. In this tutorial, we will explore the concept of thread pools in Python and how we can use them to improve the performance of our applications.

What is a Thread Pool :

• A thread pool is a collection of worker threads that are managed by a thread pool manager.
• Thread pools are useful for managing multiple tasks that need to be executed concurrently without overwhelming system resources.
• A thread pool can be used to manage a collection of threads that can be reused, instead of creating new threads every time a task is to be executed.

ThreadPoolExecutor Class in Python :

• The   ThreadPoolExecutor  class in Python is a class that provides a simple way to use a thread pool to execute multiple tasks in parallel.
• The  ThreadPoolExecutor  class can be used to execute numerous asynchronous tasks and provides methods for working with these tasks.
• The  ThreadPoolExecutor  class is a part of the concurrent.futures module in Python.

Executor Class :

• The  Executor  class is the base class for all executor implementations.
• It defines methods that can be used to submit a task to the executor, wait for tasks to be completed, and manage the executor.

Future Object :

• The Future object is a class that represents the result of a task that has been submitted to the executor.
• The Future object can be used to check if a task has completed, retrieve the result of a completed task, or cancel a task.

Submitting Multiple tasks to the thread pool :

• We can submit multiple tasks to the thread pool by two methods :
  • Using the   submit()  method
  • Using the  map()  method

Using the submit() method :

• Here, we create a thread pool with  5  threads using the  ThreadPoolExecutor  class.
• We then submit  10  tasks to the thread pool using the  submit()  method.
• Each task simply prints a message to the console, sleeps for  1  second, and then prints another message.
• Code Sample : 

from concurrent.futures import ThreadPoolExecutor
import time

def task(num):
   print("Task %s started." % num)
   time.sleep(1)
   print("Task %s completed." % num)

# Create a thread pool with 5 threads
with ThreadPoolExecutor(max_workers=5) as executor:
   # Submit 10 tasks to the thread pool
   for i in range(1, 11):
       executor.submit(task, i)
       
       
       

Using the map() method example:

• Here, we create a thread pool with 5 threads using the  ThreadPoolExecutor  class.
• We then use the  map()  method to submit multiple tasks and retrieve the results.
• Each task returns a value which is doubled and the final result is printed to the console.
• Code Sample : 

from concurrent.futures import ThreadPoolExecutor
import time
def task(num):
   print("Task %s started." % num)
   time.sleep(1)
   print("Task %s completed." % num)
   return num * 2
# Create a thread pool with 5 threads
with ThreadPoolExecutor(max_workers=5) as executor:
   # Use the map function to submit multiple tasks and retrieve the results
   results = list(executor.map(task, range(1, 11)))
print("Final result:", results)

Python ThreadPoolExecutor Practical Example :

• Here, we create a thread pool with  2   threads using the   ThreadPoolExecutor  class.
• We then download each URL in the list using the  download()  function and the thread pool.
• Finally, we print the size of each downloaded webpage to the console.
• Code Sample : 

from concurrent.futures import ThreadPoolExecutor

import requests

def download(url):
   print("Downloading %s" % url)
   response = requests.get(url)
   return response.content

# List of URLs to download
urls = ["https://www.google.com", "https://www.facebook.com", "https://www.youtube.com", "https://www.wikipedia.org"]

# Create a thread pool with 2 threads
with ThreadPoolExecutor(max_workers=2) as executor:
   # Download each URL using the thread pool
   results = executor.map(download, urls)

# Print the size of each response
for result in results:
   print("Size of response:", len(result))